Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hydra integration #39

Merged
merged 38 commits into from
Apr 9, 2024
Merged
Changes from 1 commit
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
11ca362
added function to convert units
annajungbluth Mar 19, 2024
d762b2c
added function to convert units
annajungbluth Mar 19, 2024
e5ddbac
added goes16 downloader for pipeline
annajungbluth Mar 19, 2024
765517e
added downloader for terra
annajungbluth Mar 19, 2024
d7a3323
tested terra aqua downloader
annajungbluth Mar 19, 2024
637d3f8
tested goes downloader
annajungbluth Mar 19, 2024
23fcfb0
added and tested msg downloader
annajungbluth Mar 19, 2024
49a3900
First Merge to Hydra-Integration (#35)
jejjohnson Mar 20, 2024
e4925c7
added GOES filename parser
annajungbluth Mar 21, 2024
3849a45
added msg filename parser
annajungbluth Mar 21, 2024
968e9ef
added filename parser for msg
annajungbluth Mar 21, 2024
4402f04
started developing msg pipeline
annajungbluth Mar 21, 2024
0a72a6d
continued developing msg pipeline
annajungbluth Mar 21, 2024
cee8a13
continued working on msg pipeline
annajungbluth Mar 22, 2024
7fa2144
continued hydra integration and finished download components
annajungbluth Mar 22, 2024
6ecc687
added examples to main.py
annajungbluth Mar 22, 2024
75d0db6
added goes geoprocessor to repo, added docstrings
annajungbluth Mar 27, 2024
2828f05
deleted obsolete scripts
annajungbluth Mar 27, 2024
3089ef0
tested goes geoprocessor
annajungbluth Mar 27, 2024
0618df4
updated default args
annajungbluth Mar 27, 2024
a5c3928
notebooks/dev/goes/1.4-GOES-geoprocess-val.ipynb
annajungbluth Mar 27, 2024
cad391b
tested goes geoprocessor
annajungbluth Mar 27, 2024
ef9fa9a
renamed script for consistency
annajungbluth Mar 28, 2024
4ffdf1f
work in progress modis geoprocessing
annajungbluth Mar 28, 2024
2b33e92
merged with other branch
annajungbluth Mar 28, 2024
951d280
continued developing modis geoprocessor
annajungbluth Mar 28, 2024
ba94f25
tested modis geoprocessor
annajungbluth Mar 28, 2024
a04d71e
fixed errors, standardized code, and started with msg geoprocessor
annajungbluth Apr 1, 2024
def165a
fixed modis problem
annajungbluth Apr 1, 2024
075758d
fixed modis naming issue
annajungbluth Apr 2, 2024
34f04d9
fixed small errors and tested MSG geoprocessor
annajungbluth Apr 2, 2024
0729507
fixed default arguments
annajungbluth Apr 2, 2024
10a0314
finished hydra integration of geoprocessing
annajungbluth Apr 4, 2024
7d814d2
started developing prepatcher
annajungbluth Apr 4, 2024
5eb9175
work in progress patcher
annajungbluth Apr 4, 2024
7953222
fixed coordinate, dimension problem
annajungbluth Apr 5, 2024
31b0366
tested patcher and fixed small errors
annajungbluth Apr 5, 2024
4b95a40
integrated patcher into hydra pipeline
annajungbluth Apr 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
notebooks/dev/goes/1.4-GOES-geoprocess-val.ipynb
annajungbluth committed Mar 27, 2024
commit a5c39286da08a03057f2b1abf28cea2c7f14727b
611 changes: 0 additions & 611 deletions notebooks/dev/goes/0.3-GOES-filename-parser.ipynb
Original file line number Diff line number Diff line change
@@ -261,617 +261,6 @@
"source": [
"GOESFileName.from_filename(filenames[10])"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"ds = xr.open_dataset(filenames[10])"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div><svg style=\"position: absolute; width: 0; height: 0; overflow: hidden\">\n",
"<defs>\n",
"<symbol id=\"icon-database\" viewBox=\"0 0 32 32\">\n",
"<path d=\"M16 0c-8.837 0-16 2.239-16 5v4c0 2.761 7.163 5 16 5s16-2.239 16-5v-4c0-2.761-7.163-5-16-5z\"></path>\n",
"<path d=\"M16 17c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z\"></path>\n",
"<path d=\"M16 26c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z\"></path>\n",
"</symbol>\n",
"<symbol id=\"icon-file-text2\" viewBox=\"0 0 32 32\">\n",
"<path d=\"M28.681 7.159c-0.694-0.947-1.662-2.053-2.724-3.116s-2.169-2.030-3.116-2.724c-1.612-1.182-2.393-1.319-2.841-1.319h-15.5c-1.378 0-2.5 1.121-2.5 2.5v27c0 1.378 1.122 2.5 2.5 2.5h23c1.378 0 2.5-1.122 2.5-2.5v-19.5c0-0.448-0.137-1.23-1.319-2.841zM24.543 5.457c0.959 0.959 1.712 1.825 2.268 2.543h-4.811v-4.811c0.718 0.556 1.584 1.309 2.543 2.268zM28 29.5c0 0.271-0.229 0.5-0.5 0.5h-23c-0.271 0-0.5-0.229-0.5-0.5v-27c0-0.271 0.229-0.5 0.5-0.5 0 0 15.499-0 15.5 0v7c0 0.552 0.448 1 1 1h7v19.5z\"></path>\n",
"<path d=\"M23 26h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
"<path d=\"M23 22h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
"<path d=\"M23 18h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
"</symbol>\n",
"</defs>\n",
"</svg>\n",
"<style>/* CSS stylesheet for displaying xarray objects in jupyterlab.\n",
" *\n",
" */\n",
"\n",
":root {\n",
" --xr-font-color0: var(--jp-content-font-color0, rgba(0, 0, 0, 1));\n",
" --xr-font-color2: var(--jp-content-font-color2, rgba(0, 0, 0, 0.54));\n",
" --xr-font-color3: var(--jp-content-font-color3, rgba(0, 0, 0, 0.38));\n",
" --xr-border-color: var(--jp-border-color2, #e0e0e0);\n",
" --xr-disabled-color: var(--jp-layout-color3, #bdbdbd);\n",
" --xr-background-color: var(--jp-layout-color0, white);\n",
" --xr-background-color-row-even: var(--jp-layout-color1, white);\n",
" --xr-background-color-row-odd: var(--jp-layout-color2, #eeeeee);\n",
"}\n",
"\n",
"html[theme=dark],\n",
"body[data-theme=dark],\n",
"body.vscode-dark {\n",
" --xr-font-color0: rgba(255, 255, 255, 1);\n",
" --xr-font-color2: rgba(255, 255, 255, 0.54);\n",
" --xr-font-color3: rgba(255, 255, 255, 0.38);\n",
" --xr-border-color: #1F1F1F;\n",
" --xr-disabled-color: #515151;\n",
" --xr-background-color: #111111;\n",
" --xr-background-color-row-even: #111111;\n",
" --xr-background-color-row-odd: #313131;\n",
"}\n",
"\n",
".xr-wrap {\n",
" display: block !important;\n",
" min-width: 300px;\n",
" max-width: 700px;\n",
"}\n",
"\n",
".xr-text-repr-fallback {\n",
" /* fallback to plain text repr when CSS is not injected (untrusted notebook) */\n",
" display: none;\n",
"}\n",
"\n",
".xr-header {\n",
" padding-top: 6px;\n",
" padding-bottom: 6px;\n",
" margin-bottom: 4px;\n",
" border-bottom: solid 1px var(--xr-border-color);\n",
"}\n",
"\n",
".xr-header > div,\n",
".xr-header > ul {\n",
" display: inline;\n",
" margin-top: 0;\n",
" margin-bottom: 0;\n",
"}\n",
"\n",
".xr-obj-type,\n",
".xr-array-name {\n",
" margin-left: 2px;\n",
" margin-right: 10px;\n",
"}\n",
"\n",
".xr-obj-type {\n",
" color: var(--xr-font-color2);\n",
"}\n",
"\n",
".xr-sections {\n",
" padding-left: 0 !important;\n",
" display: grid;\n",
" grid-template-columns: 150px auto auto 1fr 20px 20px;\n",
"}\n",
"\n",
".xr-section-item {\n",
" display: contents;\n",
"}\n",
"\n",
".xr-section-item input {\n",
" display: none;\n",
"}\n",
"\n",
".xr-section-item input + label {\n",
" color: var(--xr-disabled-color);\n",
"}\n",
"\n",
".xr-section-item input:enabled + label {\n",
" cursor: pointer;\n",
" color: var(--xr-font-color2);\n",
"}\n",
"\n",
".xr-section-item input:enabled + label:hover {\n",
" color: var(--xr-font-color0);\n",
"}\n",
"\n",
".xr-section-summary {\n",
" grid-column: 1;\n",
" color: var(--xr-font-color2);\n",
" font-weight: 500;\n",
"}\n",
"\n",
".xr-section-summary > span {\n",
" display: inline-block;\n",
" padding-left: 0.5em;\n",
"}\n",
"\n",
".xr-section-summary-in:disabled + label {\n",
" color: var(--xr-font-color2);\n",
"}\n",
"\n",
".xr-section-summary-in + label:before {\n",
" display: inline-block;\n",
" content: '►';\n",
" font-size: 11px;\n",
" width: 15px;\n",
" text-align: center;\n",
"}\n",
"\n",
".xr-section-summary-in:disabled + label:before {\n",
" color: var(--xr-disabled-color);\n",
"}\n",
"\n",
".xr-section-summary-in:checked + label:before {\n",
" content: '▼';\n",
"}\n",
"\n",
".xr-section-summary-in:checked + label > span {\n",
" display: none;\n",
"}\n",
"\n",
".xr-section-summary,\n",
".xr-section-inline-details {\n",
" padding-top: 4px;\n",
" padding-bottom: 4px;\n",
"}\n",
"\n",
".xr-section-inline-details {\n",
" grid-column: 2 / -1;\n",
"}\n",
"\n",
".xr-section-details {\n",
" display: none;\n",
" grid-column: 1 / -1;\n",
" margin-bottom: 5px;\n",
"}\n",
"\n",
".xr-section-summary-in:checked ~ .xr-section-details {\n",
" display: contents;\n",
"}\n",
"\n",
".xr-array-wrap {\n",
" grid-column: 1 / -1;\n",
" display: grid;\n",
" grid-template-columns: 20px auto;\n",
"}\n",
"\n",
".xr-array-wrap > label {\n",
" grid-column: 1;\n",
" vertical-align: top;\n",
"}\n",
"\n",
".xr-preview {\n",
" color: var(--xr-font-color3);\n",
"}\n",
"\n",
".xr-array-preview,\n",
".xr-array-data {\n",
" padding: 0 5px !important;\n",
" grid-column: 2;\n",
"}\n",
"\n",
".xr-array-data,\n",
".xr-array-in:checked ~ .xr-array-preview {\n",
" display: none;\n",
"}\n",
"\n",
".xr-array-in:checked ~ .xr-array-data,\n",
".xr-array-preview {\n",
" display: inline-block;\n",
"}\n",
"\n",
".xr-dim-list {\n",
" display: inline-block !important;\n",
" list-style: none;\n",
" padding: 0 !important;\n",
" margin: 0;\n",
"}\n",
"\n",
".xr-dim-list li {\n",
" display: inline-block;\n",
" padding: 0;\n",
" margin: 0;\n",
"}\n",
"\n",
".xr-dim-list:before {\n",
" content: '(';\n",
"}\n",
"\n",
".xr-dim-list:after {\n",
" content: ')';\n",
"}\n",
"\n",
".xr-dim-list li:not(:last-child):after {\n",
" content: ',';\n",
" padding-right: 5px;\n",
"}\n",
"\n",
".xr-has-index {\n",
" font-weight: bold;\n",
"}\n",
"\n",
".xr-var-list,\n",
".xr-var-item {\n",
" display: contents;\n",
"}\n",
"\n",
".xr-var-item > div,\n",
".xr-var-item label,\n",
".xr-var-item > .xr-var-name span {\n",
" background-color: var(--xr-background-color-row-even);\n",
" margin-bottom: 0;\n",
"}\n",
"\n",
".xr-var-item > .xr-var-name:hover span {\n",
" padding-right: 5px;\n",
"}\n",
"\n",
".xr-var-list > li:nth-child(odd) > div,\n",
".xr-var-list > li:nth-child(odd) > label,\n",
".xr-var-list > li:nth-child(odd) > .xr-var-name span {\n",
" background-color: var(--xr-background-color-row-odd);\n",
"}\n",
"\n",
".xr-var-name {\n",
" grid-column: 1;\n",
"}\n",
"\n",
".xr-var-dims {\n",
" grid-column: 2;\n",
"}\n",
"\n",
".xr-var-dtype {\n",
" grid-column: 3;\n",
" text-align: right;\n",
" color: var(--xr-font-color2);\n",
"}\n",
"\n",
".xr-var-preview {\n",
" grid-column: 4;\n",
"}\n",
"\n",
".xr-index-preview {\n",
" grid-column: 2 / 5;\n",
" color: var(--xr-font-color2);\n",
"}\n",
"\n",
".xr-var-name,\n",
".xr-var-dims,\n",
".xr-var-dtype,\n",
".xr-preview,\n",
".xr-attrs dt {\n",
" white-space: nowrap;\n",
" overflow: hidden;\n",
" text-overflow: ellipsis;\n",
" padding-right: 10px;\n",
"}\n",
"\n",
".xr-var-name:hover,\n",
".xr-var-dims:hover,\n",
".xr-var-dtype:hover,\n",
".xr-attrs dt:hover {\n",
" overflow: visible;\n",
" width: auto;\n",
" z-index: 1;\n",
"}\n",
"\n",
".xr-var-attrs,\n",
".xr-var-data,\n",
".xr-index-data {\n",
" display: none;\n",
" background-color: var(--xr-background-color) !important;\n",
" padding-bottom: 5px !important;\n",
"}\n",
"\n",
".xr-var-attrs-in:checked ~ .xr-var-attrs,\n",
".xr-var-data-in:checked ~ .xr-var-data,\n",
".xr-index-data-in:checked ~ .xr-index-data {\n",
" display: block;\n",
"}\n",
"\n",
".xr-var-data > table {\n",
" float: right;\n",
"}\n",
"\n",
".xr-var-name span,\n",
".xr-var-data,\n",
".xr-index-name div,\n",
".xr-index-data,\n",
".xr-attrs {\n",
" padding-left: 25px !important;\n",
"}\n",
"\n",
".xr-attrs,\n",
".xr-var-attrs,\n",
".xr-var-data,\n",
".xr-index-data {\n",
" grid-column: 1 / -1;\n",
"}\n",
"\n",
"dl.xr-attrs {\n",
" padding: 0;\n",
" margin: 0;\n",
" display: grid;\n",
" grid-template-columns: 125px auto;\n",
"}\n",
"\n",
".xr-attrs dt,\n",
".xr-attrs dd {\n",
" padding: 0;\n",
" margin: 0;\n",
" float: left;\n",
" padding-right: 10px;\n",
" width: auto;\n",
"}\n",
"\n",
".xr-attrs dt {\n",
" font-weight: normal;\n",
" grid-column: 1;\n",
"}\n",
"\n",
".xr-attrs dt:hover span {\n",
" display: inline-block;\n",
" background: var(--xr-background-color);\n",
" padding-right: 10px;\n",
"}\n",
"\n",
".xr-attrs dd {\n",
" grid-column: 2;\n",
" white-space: pre-wrap;\n",
" word-break: break-all;\n",
"}\n",
"\n",
".xr-icon-database,\n",
".xr-icon-file-text2,\n",
".xr-no-icon {\n",
" display: inline-block;\n",
" vertical-align: middle;\n",
" width: 1em;\n",
" height: 1.5em !important;\n",
" stroke-width: 0;\n",
" stroke: currentColor;\n",
" fill: currentColor;\n",
"}\n",
"</style><pre class='xr-text-repr-fallback'>&lt;xarray.DataArray &#x27;band_wavelength&#x27; (band: 1)&gt;\n",
"[1 values with dtype=float32]\n",
"Coordinates:\n",
" t datetime64[ns] ...\n",
" y_image float32 ...\n",
" x_image float32 ...\n",
" band_id (band) int8 ...\n",
" band_wavelength (band) float32 ...\n",
"Dimensions without coordinates: band\n",
"Attributes:\n",
" long_name: ABI band central wavelength\n",
" standard_name: sensor_band_central_radiation_wavelength\n",
" units: um</pre><div class='xr-wrap' style='display:none'><div class='xr-header'><div class='xr-obj-type'>xarray.DataArray</div><div class='xr-array-name'>'band_wavelength'</div><ul class='xr-dim-list'><li><span>band</span>: 1</li></ul></div><ul class='xr-sections'><li class='xr-section-item'><div class='xr-array-wrap'><input id='section-f731bb08-9211-461b-adb0-62c31c89291a' class='xr-array-in' type='checkbox' checked><label for='section-f731bb08-9211-461b-adb0-62c31c89291a' title='Show/hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-array-preview xr-preview'><span>...</span></div><div class='xr-array-data'><pre>[1 values with dtype=float32]</pre></div></div></li><li class='xr-section-item'><input id='section-bdac9820-dd2d-4f7d-ba8d-8b7d393b211b' class='xr-section-summary-in' type='checkbox' checked><label for='section-bdac9820-dd2d-4f7d-ba8d-8b7d393b211b' class='xr-section-summary' >Coordinates: <span>(5)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><ul class='xr-var-list'><li class='xr-var-item'><div class='xr-var-name'><span>t</span></div><div class='xr-var-dims'>()</div><div class='xr-var-dtype'>datetime64[ns]</div><div class='xr-var-preview xr-preview'>...</div><input id='attrs-a6541f93-6c27-43c3-b769-0c77543dfcda' class='xr-var-attrs-in' type='checkbox' ><label for='attrs-a6541f93-6c27-43c3-b769-0c77543dfcda' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-685001ae-2f7a-497f-b93f-2372562fe5c1' class='xr-var-data-in' type='checkbox'><label for='data-685001ae-2f7a-497f-b93f-2372562fe5c1' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'><dt><span>long_name :</span></dt><dd>J2000 epoch mid-point between the start and end image scan in seconds</dd><dt><span>standard_name :</span></dt><dd>time</dd><dt><span>axis :</span></dt><dd>T</dd><dt><span>bounds :</span></dt><dd>time_bounds</dd></dl></div><div class='xr-var-data'><pre>[1 values with dtype=datetime64[ns]]</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span>y_image</span></div><div class='xr-var-dims'>()</div><div class='xr-var-dtype'>float32</div><div class='xr-var-preview xr-preview'>...</div><input id='attrs-7fb53ca4-97e2-47aa-b152-333e974d27e0' class='xr-var-attrs-in' type='checkbox' ><label for='attrs-7fb53ca4-97e2-47aa-b152-333e974d27e0' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-92360da5-8694-48b4-b13e-3b86d0d2a40a' class='xr-var-data-in' type='checkbox'><label for='data-92360da5-8694-48b4-b13e-3b86d0d2a40a' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'><dt><span>long_name :</span></dt><dd>GOES-R fixed grid projection y-coordinate center of image</dd><dt><span>standard_name :</span></dt><dd>projection_y_coordinate</dd><dt><span>units :</span></dt><dd>rad</dd><dt><span>axis :</span></dt><dd>Y</dd></dl></div><div class='xr-var-data'><pre>[1 values with dtype=float32]</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span>x_image</span></div><div class='xr-var-dims'>()</div><div class='xr-var-dtype'>float32</div><div class='xr-var-preview xr-preview'>...</div><input id='attrs-fbdc059e-5ec1-42ec-8e9b-1d78b25a744c' class='xr-var-attrs-in' type='checkbox' ><label for='attrs-fbdc059e-5ec1-42ec-8e9b-1d78b25a744c' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-2ae26097-9f04-41e9-9b4c-ddd151a0fd2f' class='xr-var-data-in' type='checkbox'><label for='data-2ae26097-9f04-41e9-9b4c-ddd151a0fd2f' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'><dt><span>long_name :</span></dt><dd>GOES-R fixed grid projection x-coordinate center of image</dd><dt><span>standard_name :</span></dt><dd>projection_x_coordinate</dd><dt><span>units :</span></dt><dd>rad</dd><dt><span>axis :</span></dt><dd>X</dd></dl></div><div class='xr-var-data'><pre>[1 values with dtype=float32]</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span>band_id</span></div><div class='xr-var-dims'>(band)</div><div class='xr-var-dtype'>int8</div><div class='xr-var-preview xr-preview'>...</div><input id='attrs-0408ccc7-dca4-4c5e-acc7-a685891b5cba' class='xr-var-attrs-in' type='checkbox' ><label for='attrs-0408ccc7-dca4-4c5e-acc7-a685891b5cba' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-61c826f1-6743-4ef3-975e-a275669a9e06' class='xr-var-data-in' type='checkbox'><label for='data-61c826f1-6743-4ef3-975e-a275669a9e06' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'><dt><span>long_name :</span></dt><dd>ABI band number</dd><dt><span>standard_name :</span></dt><dd>sensor_band_identifier</dd><dt><span>units :</span></dt><dd>1</dd></dl></div><div class='xr-var-data'><pre>[1 values with dtype=int8]</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span>band_wavelength</span></div><div class='xr-var-dims'>(band)</div><div class='xr-var-dtype'>float32</div><div class='xr-var-preview xr-preview'>...</div><input id='attrs-37ee02fa-6885-44f3-b215-174c4a207792' class='xr-var-attrs-in' type='checkbox' ><label for='attrs-37ee02fa-6885-44f3-b215-174c4a207792' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-8da9520b-303c-4e34-8e17-58be8cbfd5fc' class='xr-var-data-in' type='checkbox'><label for='data-8da9520b-303c-4e34-8e17-58be8cbfd5fc' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'><dt><span>long_name :</span></dt><dd>ABI band central wavelength</dd><dt><span>standard_name :</span></dt><dd>sensor_band_central_radiation_wavelength</dd><dt><span>units :</span></dt><dd>um</dd></dl></div><div class='xr-var-data'><pre>[1 values with dtype=float32]</pre></div></li></ul></div></li><li class='xr-section-item'><input id='section-81634e0c-e097-43f4-b4c3-7780c13caa68' class='xr-section-summary-in' type='checkbox' disabled ><label for='section-81634e0c-e097-43f4-b4c3-7780c13caa68' class='xr-section-summary' title='Expand/collapse section'>Indexes: <span>(0)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><ul class='xr-var-list'></ul></div></li><li class='xr-section-item'><input id='section-5effea18-e3c5-4fbc-98de-1d3f520ac8c1' class='xr-section-summary-in' type='checkbox' checked><label for='section-5effea18-e3c5-4fbc-98de-1d3f520ac8c1' class='xr-section-summary' >Attributes: <span>(3)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><dl class='xr-attrs'><dt><span>long_name :</span></dt><dd>ABI band central wavelength</dd><dt><span>standard_name :</span></dt><dd>sensor_band_central_radiation_wavelength</dd><dt><span>units :</span></dt><dd>um</dd></dl></div></li></ul></div></div>"
],
"text/plain": [
"<xarray.DataArray 'band_wavelength' (band: 1)>\n",
"[1 values with dtype=float32]\n",
"Coordinates:\n",
" t datetime64[ns] ...\n",
" y_image float32 ...\n",
" x_image float32 ...\n",
" band_id (band) int8 ...\n",
" band_wavelength (band) float32 ...\n",
"Dimensions without coordinates: band\n",
"Attributes:\n",
" long_name: ABI band central wavelength\n",
" standard_name: sensor_band_central_radiation_wavelength\n",
" units: um"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ds.band_wavelength"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/opt/anaconda3/envs/iti-env/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
" from .autonotebook import tqdm as notebook_tqdm\n"
]
}
],
"source": [
"from rs_tools._src.utils.io import get_list_filenames\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"def parse_goes16_dates_from_file(file: str):\n",
" timestamp = Path(file).name.replace(\"-\",\"_\").split(\"_\")\n",
" return datetime.strptime(timestamp[-2][1:], \"%Y%j%H%M%S%f\").strftime(\"%Y%j%H%M\")\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"def goes_files(read_path):\n",
" \"\"\"\n",
" Returns a list of all GOES files in the read path.\n",
"\n",
" Returns:\n",
" List[str]: A list of file paths.\n",
" \"\"\"\n",
" # get a list of all GOES files from specified path\n",
" files = get_list_filenames(read_path, \".nc\")\n",
" return files"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"path = '/Users/anna.jungbluth/Desktop/git/rs_tools/data/goes16/'"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"files_ = goes_files(path)\n",
"unique_times = list(set(map(parse_goes16_dates_from_file, files_)))"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['20202761419',\n",
" '20182741420',\n",
" '20202761439',\n",
" '20182751411',\n",
" '20202751409',\n",
" '20182741405',\n",
" '20202761409']"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"unique_times"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [],
"source": [
"# get files from unique times\n",
"files = []\n",
"for ut in unique_times:\n",
" files.extend(filter(lambda x: ut in x, files_))"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [],
"source": [
"files = list(filter(lambda x: unique_times[1] in x, files_))"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [],
"source": [
"files = list(filter(lambda x: \"Rad\" in x, files))"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['/Users/anna.jungbluth/Desktop/git/rs_tools/data/goes16/L1b/noaa-goes16/ABI-L1b-RadF/2018/274/14/OR_ABI-L1b-RadF-M4C01_G16_s20182741415224_e20182741420026_c20182741420080.nc',\n",
" '/Users/anna.jungbluth/Desktop/git/rs_tools/data/goes16/L1b/noaa-goes16/ABI-L1b-RadF/2018/274/14/OR_ABI-L1b-RadF-M4C02_G16_s20182741415224_e20182741420026_c20182741420075.nc',\n",
" '/Users/anna.jungbluth/Desktop/git/rs_tools/data/goes16/L1b/noaa-goes16/ABI-L1b-RadF/2018/274/14/OR_ABI-L1b-RadF-M4C03_G16_s20182741415224_e20182741420026_c20182741420081.nc',\n",
" '/Users/anna.jungbluth/Desktop/git/rs_tools/data/goes16/L1b/noaa-goes16/ABI-L1b-RadF/2018/274/14/OR_ABI-L1b-RadF-M4C04_G16_s20182741415224_e20182741420027_c20182741420080.nc',\n",
" '/Users/anna.jungbluth/Desktop/git/rs_tools/data/goes16/L1b/noaa-goes16/ABI-L1b-RadF/2018/274/14/OR_ABI-L1b-RadF-M4C05_G16_s20182741415224_e20182741420026_c20182741420082.nc',\n",
" '/Users/anna.jungbluth/Desktop/git/rs_tools/data/goes16/L1b/noaa-goes16/ABI-L1b-RadF/2018/274/14/OR_ABI-L1b-RadF-M4C06_G16_s20182741415224_e20182741420033_c20182741420067.nc',\n",
" '/Users/anna.jungbluth/Desktop/git/rs_tools/data/goes16/L1b/noaa-goes16/ABI-L1b-RadF/2018/274/14/OR_ABI-L1b-RadF-M4C07_G16_s20182741415224_e20182741420039_c20182741420083.nc',\n",
" '/Users/anna.jungbluth/Desktop/git/rs_tools/data/goes16/L1b/noaa-goes16/ABI-L1b-RadF/2018/274/14/OR_ABI-L1b-RadF-M4C08_G16_s20182741415224_e20182741420027_c20182741420083.nc',\n",
" '/Users/anna.jungbluth/Desktop/git/rs_tools/data/goes16/L1b/noaa-goes16/ABI-L1b-RadF/2018/274/14/OR_ABI-L1b-RadF-M4C09_G16_s20182741415224_e20182741420033_c20182741420101.nc',\n",
" '/Users/anna.jungbluth/Desktop/git/rs_tools/data/goes16/L1b/noaa-goes16/ABI-L1b-RadF/2018/274/14/OR_ABI-L1b-RadF-M4C10_G16_s20182741415224_e20182741420039_c20182741420095.nc',\n",
" '/Users/anna.jungbluth/Desktop/git/rs_tools/data/goes16/L1b/noaa-goes16/ABI-L1b-RadF/2018/274/14/OR_ABI-L1b-RadF-M4C11_G16_s20182741415224_e20182741420027_c20182741420096.nc',\n",
" '/Users/anna.jungbluth/Desktop/git/rs_tools/data/goes16/L1b/noaa-goes16/ABI-L1b-RadF/2018/274/14/OR_ABI-L1b-RadF-M4C12_G16_s20182741415224_e20182741420033_c20182741420094.nc',\n",
" '/Users/anna.jungbluth/Desktop/git/rs_tools/data/goes16/L1b/noaa-goes16/ABI-L1b-RadF/2018/274/14/OR_ABI-L1b-RadF-M4C13_G16_s20182741415224_e20182741420039_c20182741420100.nc',\n",
" '/Users/anna.jungbluth/Desktop/git/rs_tools/data/goes16/L1b/noaa-goes16/ABI-L1b-RadF/2018/274/14/OR_ABI-L1b-RadF-M4C14_G16_s20182741415224_e20182741420026_c20182741420101.nc',\n",
" '/Users/anna.jungbluth/Desktop/git/rs_tools/data/goes16/L1b/noaa-goes16/ABI-L1b-RadF/2018/274/14/OR_ABI-L1b-RadF-M4C15_G16_s20182741415224_e20182741420032_c20182741420102.nc',\n",
" '/Users/anna.jungbluth/Desktop/git/rs_tools/data/goes16/L1b/noaa-goes16/ABI-L1b-RadF/2018/274/14/OR_ABI-L1b-RadF-M4C16_G16_s20182741415224_e20182741420038_c20182741420099.nc']"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"files"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'long_name': 'ABI L1b Radiances',\n",
" 'standard_name': 'toa_outgoing_radiance_per_unit_wavelength',\n",
" 'sensor_band_bit_depth': 10,\n",
" 'valid_range': array([ 0, 1022], dtype=int16),\n",
" 'units': 'W m-2 sr-1 um-1',\n",
" 'resolution': 'y: 0.000028 rad x: 0.000028 rad',\n",
" 'grid_mapping': 'goes_imager_projection',\n",
" 'cell_methods': 't: point area: point',\n",
" 'ancillary_variables': 'DQF'}"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ds.Rad.attrs"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
120 changes: 75 additions & 45 deletions rs_tools/_src/geoprocessing/goes/geoprocessor_goes.py
Original file line number Diff line number Diff line change
@@ -15,6 +15,7 @@
from rs_tools._src.geoprocessing.goes.validation import correct_goes16_bands, correct_goes16_satheight
from rs_tools._src.geoprocessing.goes.reproject import add_goes16_crs
from rs_tools._src.geoprocessing.reproject import convert_lat_lon_to_x_y, calc_latlon
from rs_tools._src.geoprocessing.utils import check_sat_FOV
import pandas as pd
from datetime import datetime
from functools import partial
@@ -53,7 +54,7 @@ class GOES16GeoProcessing:
resolution: float
read_path: str
save_path: str
region: Tuple[int, int, int, int]
region: Optional[Tuple[int, int, int, int]]
resample_method: str

@property
@@ -91,17 +92,27 @@ def preprocess_fn(self, ds: xr.Dataset) -> Tuple[xr.Dataset, xr.Dataset]:
# assign coordinate reference system
ds = add_goes16_crs(ds)

if self.region is not None:
if self.region != (None, None, None, None):
logger.info(f"Subsetting data to region: {self.region}")
# subset data
lon_bnds = (self.region[0], self.region[2])
lat_bnds = (self.region[1], self.region[3])
# convert lat lon bounds to x y (in meters)
x_bnds, y_bnds = convert_lat_lon_to_x_y(ds.FOV.crs, lon=lon_bnds, lat=lat_bnds, )
# check that region is within the satellite field of view
# compile satellite FOV
satellite_FOV = (min(ds.x.values), min(ds.y.values), max(ds.x.values), max(ds.y.values))
# compile region bounds in x y
region_xy = (x_bnds[0], y_bnds[0], x_bnds[1], y_bnds[1])
if not check_sat_FOV(region_xy, FOV=satellite_FOV):
raise ValueError("Region is not within the satellite field of view")

ds = ds.sortby("x").sortby("y")
# slice based on x y bounds
ds = ds.sel(y=slice(y_bnds[0], y_bnds[1]), x=slice(x_bnds[0], x_bnds[1]))

if self.resolution is not None:
logger.info(f"Resampling data to resolution: {self.resolution} m")
# resampling
ds_subset = resample_rioxarray(ds, resolution=self.resolution, method=self.resample_method)
else:
@@ -171,46 +182,6 @@ def preprocess_fn_cloudmask(self, ds: xr.Dataset) -> xr.Dataset:

return ds_subset

def preprocess_files(self):
"""
Preprocesses multiple files in the read path and saves processed files to the save path.
"""
# get unique times from read path
unique_times = list(set(map(parse_goes16_dates_from_file, self.goes_files)))

pbar_time = tqdm(unique_times)

for itime in pbar_time:

pbar_time.set_description(f"Processing: {itime}")

# get files from unique times
files = list(filter(lambda x: itime in x, self.goes_files))

try:
# load radiances
ds = self.preprocess_radiances(files)
except AssertionError:
logger.error(f"Skipping {itime} due to missing bands")
continue
try:
# load cloud mask
ds_clouds = self.preprocess_cloud_mask(files)["cloud_mask"]
except AssertionError:
logger.error(f"Skipping {itime} due to missing cloud mask")
continue

# interpolate cloud mask to data
ds_clouds = ds_clouds.interp(x=ds.x, y=ds.y)
# save cloud mask as data coordinate
ds = ds.assign_coords({"cloud_mask": (("y", "x"), ds_clouds.values.squeeze())})
ds["cloud_mask"].attrs = ds_clouds.attrs

if not os.path.exists(self.save_path):
os.makedirs(self.save_path)

ds.to_netcdf(Path(self.save_path).joinpath(f"{itime}_goes16.nc"), engine="netcdf4")

def preprocess_radiances(self, files: List[str]) -> xr.Dataset:
"""
Preprocesses radiances from the input files.
@@ -282,21 +253,69 @@ def preprocess_cloud_mask(self, files: List[str]) -> xr.Dataset:

return ds

def preprocess_files(self):
"""
Preprocesses multiple files in the read path and saves processed files to the save path.
"""
# get unique times from read path
unique_times = list(set(map(parse_goes16_dates_from_file, self.goes_files)))

pbar_time = tqdm(unique_times)

for itime in pbar_time:

pbar_time.set_description(f"Processing: {itime}")

# get files from unique times
files = list(filter(lambda x: itime in x, self.goes_files))

try:
# load radiances
ds = self.preprocess_radiances(files)
except AssertionError:
logger.error(f"Skipping {itime} due to missing bands")
continue
try:
# load cloud mask
ds_clouds = self.preprocess_cloud_mask(files)["cloud_mask"]
except AssertionError:
logger.error(f"Skipping {itime} due to missing cloud mask")
continue

# interpolate cloud mask to data
ds_clouds = ds_clouds.interp(x=ds.x, y=ds.y)
# save cloud mask as data coordinate
ds = ds.assign_coords({"cloud_mask": (("y", "x"), ds_clouds.values.squeeze())})
ds["cloud_mask"].attrs = ds_clouds.attrs

# check if save path exists, and create if not
if not os.path.exists(self.save_path):
os.makedirs(self.save_path)

# remove file if it already exists
save_filename = Path(self.save_path).joinpath(f"{itime}_goes16.nc")
if os.path.exists(save_filename):
logger.info(f"File already exists. Overwriting file: {save_filename}")
os.remove(save_filename)
# save to netcdf
ds.to_netcdf(save_filename, engine="netcdf4")


def geoprocess_goes16(
resolution: float = None, # defined in meters
read_path: str = "./",
save_path: str = "./",
region: Tuple[int, int, int, int] = (-130, -15, -90, 5),
region: Tuple[int, int, int, int] = (None, None, None, None),
resample_method: str = "bilinear",
):
"""
Downloads MODIS TERRA and GOES 16 files for the specified period, region, and save path.
Args:
resolution (float, optional): The resolution of the downloaded files in meters. Defaults to 1_000.
resolution (float, optional): The resolution of the downloaded files in meters. Defaults to None.
read_path (str, optional): The path to read the files from. Defaults to "./".
save_path (str, optional): The path to save the downloaded files. Defaults to "./".
region (Tuple[int, int, int, int], optional): The geographic region to download files for. Defaults to (-130, -15, -90, 5).
region (Tuple[int, int, int, int], optional): The geographic region to download files for. Defaults to None.
resample_method (str, optional): The resampling method to use. Defaults to "bilinear".
Returns:
@@ -319,6 +338,17 @@ def geoprocess_goes16(

if __name__ == '__main__':
"""
# =========================
# Test Cases
# =========================
python geoprocessor_goes.py --read-path "/home/data" --save-path /home/data/goes/geoprocessed
python geoprocessor_goes.py --read-path "/home/data" --save-path /home/data/goes/geoprocessed --resolution 1000
python geoprocessor_goes.py --read-path "/home/data" --save-path /home/data/goes/geoprocessed --resolution 5000
python geoprocessor_goes.py --read-path "/home/data" --save-path /home/data/goes/geoprocessed --resolution 5000 --region -130 -15 -90 5
# ====================
# FAILURE TEST CASES
# ====================
python geoprocessor_goes.py --read-path "/home/data" --save-path /home/data/goes/geoprocessed --resolution 5000 --region -200 -15 90 5
"""
typer.run(geoprocess_goes16)
19 changes: 19 additions & 0 deletions rs_tools/_src/geoprocessing/utils.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
import numpy as np
from typing import Tuple

def check_sat_FOV(region: Tuple[int, int, int, int], FOV: Tuple[int, int, int, int]) -> bool:
"""
Check if the region is within the Field of View (FOV) of the satellite.
Args:
region (Tuple[int, int, int, int]): The region (lon_min, lat_min, lon_max, lat_max) to check if it is within the FOV
FOV (Tuple[int, int]): The Field of View (FOV) (lon_min, lat_min, lon_max, lat_max) of the satellite.
Returns:
bool: True if the region is within the FOV, False otherwise.
"""
# Check if the region is within the Field of View (FOV) of the satellite.
if abs(region[0]) <= abs(FOV[0]) and abs(region[1]) <= abs(FOV[1]) and abs(region[2]) <= abs(FOV[2]) and abs(region[3]) <= abs(FOV[3]):
return True
else:
return False