Release v16.3.22 optimized point output #3257

AliS-Noaa · 2025-01-24T21:44:37Z

Description

following @dkokron work on the per time step output reading the ww3_outp is optimized to read per time step binary output and create per point spectral and bull/cbull output.
Addressing #2913

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? YES
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? YES (If YES, please add a link to any PRs that are pending.)
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model GFSv16 optimized point output WW3#1355 (comment)
- wxflow

How has this been tested?

This one was for operational branch and we ran similar runs as ops and got identical point outputs with less resources. Testing is conducted on WCOSS2 (Cactus). This will be incorporated into the dev branch soon.

Checklist

…ource allocatio, issue NOAA-EMC#3157

…#3157

…-v16.3.22-pntoutp

aerorahul

Just a couple things that caught my eye.
Looks like a dramatic reduction in walltimes and resources.

aerorahul · 2025-01-25T02:03:03Z

scripts/exgfs_wave_post_pnt.sh

-      ln -fs ${pfile} ./out_pnt.${waveuoutpGRD}
-    else 
+    then
+      ln -fs ${pfile}


The target name is missing here as well.

aerorahul · 2025-01-25T02:03:06Z

scripts/exgfs_wave_post_pnt.sh

@@ -252,7 +240,7 @@
    HMS="$(echo $CDATE | cut -c9-10)0000"
    if [ -f $COMIN/rundata/${WAV_MOD_TAG}.out_pnt.${waveuoutpGRD}.${YMD}.${HMS} ]
    then
-      ln -s $COMIN/rundata/${WAV_MOD_TAG}.out_pnt.${waveuoutpGRD}.${YMD}.${HMS} ./out_pnt.${waveuoutpGRD}   
+      ln -s $COMIN/rundata/${WAV_MOD_TAG}.out_pnt.${waveuoutpGRD}.${YMD}.${HMS}


The target is missing.

KateFriedman-NOAA · 2025-01-25T13:19:27Z

sorc/checkout.sh

-    git clone --recursive --branch GFS.v16.3.1 https://github.com/ufs-community/ufs-weather-model.git fv3gfs.fd >> ${topdir}/checkout-fv3gfs.log 2>&1
+    git clone --recursive --branch prodpoint https://github.com/AliS-Noaa/ufs-weather-model.git fv3gfs.fd >> ${topdir}/checkout-fv3gfs.log 2>&1
+    #git clone --recursive --branch GFS.v16.3.1 https://github.com/ufs-community/ufs-weather-model.git fv3gfs.fd >> ${topdir}/checkout-fv3gfs.log 2>&1


Leaving note to remind myself to check back on this before approving and merging.

KateFriedman-NOAA

Changes look good! I agree with @aerorahul comments. Will review and approve once the updated ufs-weather-model tag is available and in this PR, as well as updates to the release notes. Thanks @AliS-Noaa !

JessicaMeixner-NOAA · 2025-01-27T18:55:04Z

@AliS-Noaa is testing the changes based on the review and then will push them once test finish and our successful.

JessicaMeixner-NOAA · 2025-01-28T19:49:10Z

docs/Release_Notes.md

-The upstream OBSPROC package is updated to v1.3. Along with this are the following companion updates:
-* workflow and UFS_UTILS package updates to use the new AFWA global snow file due to the hemispheric snow files being phased out
-* updated GSI code and convinfo file for saildrone observations
+The WW3 point output program (ww3_outp) is improved to process per-time-step point output to reduce the resources for the wave_post_pnt jobs.


@KateFriedman-NOAA did we need to add or replace this?

The text should be added after the existing text in the prelude and the text on the prior lines should remain. It should essentially be text to say that this wave change is also being included.

JessicaMeixner-NOAA · 2025-01-28T19:50:00Z

docs/Release_Notes.md

-[email protected]
-[email protected]
-[email protected]
+Ali Salimi Tarazouj


@AliS-Noaa I think we needed to add to this, not replace and put our emails so like:

[email protected]
[email protected]
[email protected]
[email protected]
[email protected]

Yes to @JessicaMeixner-NOAA 's comment. Leave the current list of emails and add your emails to the list, just as Jessica outlined.

JessicaMeixner-NOAA · 2025-01-28T19:50:32Z

docs/Release_Notes.md


 HPSS ARCHIVE
 ------------

-* No changes from GFS v16.3.21
+* No changes from GFS v16.3.22


This is version 16.3.22, so all these "no changes from" should stay at v16.3.21

docs/Release_Notes.md

Co-authored-by: Jessica Meixner <[email protected]>

docs/Release_Notes.md

Co-authored-by: Jessica Meixner <[email protected]>

AliS-Noaa · 2025-01-28T20:28:41Z

@JessicaMeixner-NOAA Thank you!

JessicaMeixner-NOAA · 2025-01-28T21:57:46Z

The WW3 PR has been merged. The UFS model PR should be merged soon and a new tag GFS.v16.3.22 should be made likely tomorrow.

KateFriedman-NOAA · 2025-01-29T15:23:16Z

docs/Release_Notes.md

@@ -51,13 +51,14 @@ cd ../ecf
 VERSION FILE CHANGES
 --------------------

-* `versions/run.ver` - change `version=v16.3.22`, `gfs_ver=v16.3.22`, and `obsproc_ver=v1.3`
+* * `versions/run.ver` - change `version=v16.3.22`, `gfs_ver=v16.3.22`, and `obsproc_ver=v1.3`


Please remove the addition of a second asterix.

KateFriedman-NOAA · 2025-01-29T15:24:01Z

docs/Release_Notes.md

@@ -67,17 +68,17 @@ JOBS CHANGES
 PARM/CONFIG CHANGES
 -------------------

-* No changes from GFS v16.3.21
+* `config.resources.emc.dyn` and `config.resources.nco.static` are changed to reduce the resources accordingly.


Reduce the resources of what? Since this package includes multiple things now, please specify what job resources are changing.

KateFriedman-NOAA · 2025-01-29T15:24:34Z

docs/Release_Notes.md


 SCRIPT CHANGES
 --------------

-* No changes from GFS v16.3.21
+* `scripts/exgfs_wave_post_pnt.sh` is changed to create be compatible with the new `ww3_outp`.


Review wording in this comment.

KateFriedman-NOAA · 2025-01-29T15:25:46Z

docs/Release_Notes.md


 FIX CHANGES
 -----------

-* GSI `global_convinfo.txt` fix update for saildrone
+* GSI `global_convinfo.txt` fix update for saildrone```


Remove the three ` at end of line

KateFriedman-NOAA · 2025-01-29T15:26:19Z

docs/Release_Notes.md

@@ -88,23 +89,31 @@ CHANGES TO FILE AND FILE SIZES
 ------------------------------

 No longer ingest:
-* `${RUN}.${cycle}.NPR.SNWN.SP.S1200.MESH16.grb` (`AFWA_NH_FILE`)
+* `${RUN}.${cycle}.NPR.SNWN.SP.S1200.MESH16.grb` (`AFWA_NH_FILE`)	


Remove added spaces on line 92

KateFriedman-NOAA · 2025-01-29T15:26:32Z

docs/Release_Notes.md

 * `${RUN}.${cycle}.NPR.SNWS.SP.S1200.MESH16.grb` (`AFWA_SH_FILE`)

-Now ingest:
+Now ingest:	


Remove added spaces (or tab) on line 95

…ease note

AliS-Noaa added 8 commits January 10, 2025 18:28

optimized the ww3_outp, issue NOAA-EMC#3157

a2ba32b

cleaned the outp script, issue NOAA-EMC#3157

8086f49

changing the point output script and cleaning it and changing the res…

84fc8f3

…ource allocatio, issue NOAA-EMC#3157

concising the exgfs_wave_post_pnt.sh and config files, issue NOAA-EMC…

d2b6b70

…#3157

fixing tar script for the optimized point output, issue NOAA-EMC#3157

d122080

retreiving the file counting, issue NOAA-EMC#3157

8e2ad85

Merge remote-tracking branch 'origin/devgfs/pointupdate' into release…

0772d72

…-v16.3.22-pntoutp

Revising the checkout.sh

1a9b7f9

aerorahul reviewed Jan 25, 2025

View reviewed changes

KateFriedman-NOAA self-requested a review January 25, 2025 13:17

KateFriedman-NOAA reviewed Jan 25, 2025

View reviewed changes

fixing the links to inputs for wave post script

5784053

AliS-Noaa mentioned this pull request Jan 28, 2025

[production/GFS.v16] GFSv16 prod update of WW3 for optimized point output ufs-community/ufs-weather-model#2575

Merged

15 tasks

WalterKolczynski-NOAA added the production update Processing update in production label Jan 28, 2025

Updatuing the Release_Notes.md for wave point output update

14059bf

JessicaMeixner-NOAA reviewed Jan 28, 2025

View reviewed changes

AliS-Noaa added 2 commits January 28, 2025 19:57

minor revision in the Release_Notes.md

4a2c51b

another minor correction to the Release_Notes.md

25f6086

JessicaMeixner-NOAA reviewed Jan 28, 2025

View reviewed changes

docs/Release_Notes.md Outdated Show resolved Hide resolved

Update docs/Release_Notes.md

53f8172

Co-authored-by: Jessica Meixner <[email protected]>

JessicaMeixner-NOAA reviewed Jan 28, 2025

View reviewed changes

AliS-Noaa and others added 6 commits January 28, 2025 15:26

Update docs/Release_Notes.md

61e3577

Co-authored-by: Jessica Meixner <[email protected]>

Update docs/Release_Notes.md

0892513

Co-authored-by: Jessica Meixner <[email protected]>

Update docs/Release_Notes.md

b2220cc

Co-authored-by: Jessica Meixner <[email protected]>

Update docs/Release_Notes.md

52558ac

Co-authored-by: Jessica Meixner <[email protected]>

Update docs/Release_Notes.md

e22af6b

Co-authored-by: Jessica Meixner <[email protected]>

Update docs/Release_Notes.md

aaf1a46

Co-authored-by: Jessica Meixner <[email protected]>

AliS-Noaa added 2 commits January 28, 2025 22:11

adding two wave jobs for testing to the Release note

de078f6

adding sperate log file for each wave post point jobs

14bb68f

KateFriedman-NOAA reviewed Jan 29, 2025

View reviewed changes

adding the tag to checkout.sh and also some minor revision of the rel…

9ae2e74

…ease note

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release v16.3.22 optimized point output #3257

Release v16.3.22 optimized point output #3257

AliS-Noaa commented Jan 24, 2025

aerorahul left a comment

aerorahul Jan 25, 2025

aerorahul Jan 25, 2025

KateFriedman-NOAA Jan 25, 2025

KateFriedman-NOAA left a comment •

edited

Loading

JessicaMeixner-NOAA commented Jan 27, 2025

JessicaMeixner-NOAA Jan 28, 2025

KateFriedman-NOAA Jan 28, 2025

JessicaMeixner-NOAA Jan 28, 2025

KateFriedman-NOAA Jan 28, 2025

JessicaMeixner-NOAA Jan 28, 2025

KateFriedman-NOAA Jan 28, 2025

AliS-Noaa commented Jan 28, 2025

JessicaMeixner-NOAA commented Jan 28, 2025

KateFriedman-NOAA Jan 29, 2025

KateFriedman-NOAA Jan 29, 2025

KateFriedman-NOAA Jan 29, 2025

KateFriedman-NOAA Jan 29, 2025 •

edited

Loading

KateFriedman-NOAA Jan 29, 2025

KateFriedman-NOAA Jan 29, 2025

Release v16.3.22 optimized point output #3257

Are you sure you want to change the base?

Release v16.3.22 optimized point output #3257

Conversation

AliS-Noaa commented Jan 24, 2025

Description

Type of change

Change characteristics

How has this been tested?

Checklist

aerorahul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KateFriedman-NOAA left a comment • edited Loading

Choose a reason for hiding this comment

JessicaMeixner-NOAA commented Jan 27, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AliS-Noaa commented Jan 28, 2025

JessicaMeixner-NOAA commented Jan 28, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KateFriedman-NOAA Jan 29, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KateFriedman-NOAA left a comment •

edited

Loading

KateFriedman-NOAA Jan 29, 2025 •

edited

Loading