Update scripts to use the GW-RT global namelist #3017

dpsarmie · 2024-10-18T18:45:49Z

Description

The following is a Google Doc with new variables that were added to the nml template with this PR: Google Sheets link

This PR changes the ways the input.nml file is created. Instead of creating a namelist, the variables that are set locally will now use the UFS global_control.nml.IN template and the atparse function to generate the namelist.

This should allow the GW branch to more easily test changes in the UFS regression test framework. It will also allow GW to more easily introduce UFS changes into GW.

The functionality in parsing_namelists_FV3.sh is mostly unchanged. Since variables are still saved as local, the parsing of the template will occur in this script. Sections in the namelist that were previously not generated (stoch physics options for example) will be set and parsed since the regression test template has these placeholders active. As options are turned off and on, ! will be used in the namelist to comment out sections that are not used.

Previous PR #2733 was closed since my old branch was stale and a new branch was created.

Unresolved issues and input needed from GW group before final code review

First of all, global_control.nml.IN has been added in the root directory of global_workflow. This will not be in the final commit. This is a place holder until the pathing to the UFS template is finalized. A variable with the final path will also be needed to replace the hardcoded path in parsing_namelists_FV3.sh.

There are some variables that were not set in the original scripts that have been added. These were set to the UFS defaults.
The tests I've conducted were to generate the namelists but more tests should be done to make sure that outputs have not changed.

Resolves #2731
Resolves ufs-community/ufs-weather-model#1664

Outstanding dependencies:
ufs-community/ufs-weather-model#2425

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? YES
Does this change require a documentation update? YES
Does this change require an update to any of the following submodules? YES
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model GW-RT global_control.nml modifications ufs-community/ufs-weather-model#2425
- wxflow

How has this been tested?

Tested namelist generation with C48ATM and C48S2S configurations. Further testing will need to be done to ensure intended funcitonality.

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have documented my code, including function, input, and output descriptions
My changes generate no new warnings
New and existing tests pass with my changes
This change is covered by an existing CI test or a new one has been added
I have made corresponding changes to the system documentation if necessary

WalterKolczynski-NOAA · 2024-10-24T20:01:16Z

Not a fan of the hiding, at least not how it is implemented here. All the variables are still set by the parsing script, it makes the template more difficult to read, and doesn't even remove it from the rendered namelist, just comments the lines out. Given that, I'd rather just leave all the settings in the namelist with the values that turn them off.

dpsarmie · 2024-10-25T18:27:53Z

Hey Walter,
Agreed, this wasn't my first choice but I didn't want to completely overhaul the scripts on the first go.
We could think about also integrating the default_vars.sh script into this workflow, but I didn't know how you all would feel about saving the environmental variables globally vs locally. If it is allowed, then you could in theory run the default_vars.sh script to set any variables that aren't defined by the user's previous scripts before populating the namelist. That should cut down on the amount of bulk in the parsing_fv3 script.

Splitting up the sections into different scripts could also work, but you run into the issue of not being able to easily pass variables between scripts if they are saved as locals (correct me if I'm wrong here).

Also open to your suggestions on how to best make this fit within the GW system.

dpsarmie · 2024-10-28T18:49:48Z

Here is a doc with the differences between variable values for GW and RT for a C48 atm only case: GW-RT differences

We're leaning towards just changing the RT baselines to match what GW has, but we can discuss this if needed.

dpsarmie · 2024-10-30T18:30:10Z

ush/parsing_namelists_FV3.sh

+local NPZP=${LEVS} #levp
+local GFS_DWINDS=${gfs_dwinds}
+
+local FHZERO=${FHZER}


Old script saved this as FHZER (ush/parsing_namelists_FV3.sh L196). Kept the convention for now.

We should just update the other instances:

parm/config/gfs/config.efcs:export FHZER=6 parm/config/gefs/config.fcst:export FHZER=@FHZER@ parm/config/gefs/yaml/defaults.yaml: FHZER: 6 ush/parsing_namelists_FV3_nest.sh: fhzero = ${FHZER}

WalterKolczynski-NOAA

Structurally it looks fine. Didn't verify that the default for every individual setting is unchanged. Just take care of the FHZER thing and I'm happy with this.

WalterKolczynski-NOAA · 2025-01-08T04:38:42Z

ush/parsing_namelists_FV3.sh

+local NPZP=${LEVS} #levp
+local GFS_DWINDS=${gfs_dwinds}
+
+local FHZERO=${FHZER}


We should just update the other instances:

parm/config/gfs/config.efcs:export FHZER=6 parm/config/gefs/config.fcst:export FHZER=@FHZER@ parm/config/gefs/yaml/defaults.yaml: FHZER: 6 ush/parsing_namelists_FV3_nest.sh: fhzero = ${FHZER}

WalterKolczynski-NOAA · 2025-01-16T06:12:07Z

@dpsarmie Merge in develop again and I think this is ready for CI

dpsarmie · 2025-01-16T14:53:10Z

I was still running tests on my end as a final check but should be done before lunch. We should still talk about the final pathing to the template (just added a review for visibility).

Added a missing period

Single quotes have been forced on all filenames in the namelist. PERT_CLDS has also been set to .false. unless the DO_SPPT flag is on, then PRET_CLDS is set to .true.

emcbot · 2025-01-22T04:21:08Z

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 2 with error logs:

/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_860102e8/logs/2021032418/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-01-22T04:21:12Z

Experiment C96_atm3DVar FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/EXPDIR/C96_atm3DVar_860102e8

emcbot · 2025-01-22T04:21:13Z

Experiment C96_S2SWA_gefs_replay_ics FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/EXPDIR/C96_S2SWA_gefs_replay_ics_860102e8

emcbot · 2025-01-22T04:21:19Z

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_860102e8

emcbot · 2025-01-22T04:21:20Z

Experiment C48_S2SWA_gefs FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/EXPDIR/C48_S2SWA_gefs_860102e8

emcbot · 2025-01-22T04:21:22Z

Experiment C48mx500_hybAOWCDA FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/EXPDIR/C48mx500_hybAOWCDA_860102e8

emcbot · 2025-01-22T04:21:26Z

Experiment C96C48_hybatmDA FAILED on Hera in Build# 2 with error logs:

/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C96C48_hybatmDA_860102e8/logs/2021122018/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-01-22T04:21:26Z

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 2 with error logs:

/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_860102e8/logs/2021122012/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-01-22T04:21:27Z

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 2 with error logs:

/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_860102e8/logs/2024022318/gdas_fcst_seg0.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-01-22T04:21:34Z

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_860102e8

emcbot · 2025-01-22T04:21:34Z

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_860102e8

emcbot · 2025-01-22T04:21:35Z

Experiment C96C48_hybatmDA FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/EXPDIR/C96C48_hybatmDA_860102e8

emcbot · 2025-01-22T07:29:48Z

Experiment C48_ATM FAILED on Hera in Build# 2 with error logs:

/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C48_ATM_860102e8/logs/2021032318/gfs_metpg2o1.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-01-22T07:29:55Z

Experiment C48_ATM FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/EXPDIR/C48_ATM_860102e8

emcbot · 2025-01-22T07:36:30Z

Experiment C48_S2SW FAILED on Hera in Build# 2 with error logs:

/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C48_S2SW_860102e8/logs/2021032318/gfs_metpg2o1.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-01-22T07:36:37Z

Experiment C48_S2SW FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/EXPDIR/C48_S2SW_860102e8

emcbot · 2025-01-22T07:36:47Z

CI Failed on Hera in Build# 2
Built and ran in directory /scratch1/NCEPDEV/global/CI/3017


Experiment C48mx500_3DVarAOWCDA_860102e8 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Jan 22 04:20:48 UTC 2025
Experiment C48mx500_3DVarAOWCDA_860102e8 Terminated: *FAIL*
Experiment C96_S2SWA_gefs_replay_ics_860102e8 Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Wed Jan 22 04:20:50 UTC 2025
Experiment C96_S2SWA_gefs_replay_ics_860102e8 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_860102e8/logs/2021032418/gdas_fcst_seg0.log
Experiment C96_atm3DVar_860102e8 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Jan 22 04:20:54 UTC 2025
Experiment C96_atm3DVar_860102e8 Terminated: *FAIL*
Experiment C48mx500_hybAOWCDA_860102e8 Terminated with 0
FAIL
FAIL tasks failed and 3 dead at Wed Jan 22 04:20:54 UTC 2025
Experiment C48mx500_hybAOWCDA_860102e8 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C96_S2SWA_gefs_replay_ics_860102e8/logs/2020110100/gefs_fcst_mem001_seg0.log
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C96_S2SWA_gefs_replay_ics_860102e8/logs/2020110100/gefs_fcst_mem002_seg0.log
Experiment C48_S2SWA_gefs_860102e8 Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Wed Jan 22 04:20:55 UTC 2025
Experiment C48_S2SWA_gefs_860102e8 Terminated: *FAIL*
Experiment C96C48_hybatmDA_860102e8 Terminated with 0
FAIL
FAIL tasks failed and 3 dead at Wed Jan 22 04:20:56 UTC 2025
Experiment C96C48_hybatmDA_860102e8 Terminated: *FAIL*
Experiment C96C48_hybatmaerosnowDA_860102e8 Terminated with 0
FAIL
FAIL tasks failed and 3 dead at Wed Jan 22 04:20:56 UTC 2025
Experiment C96C48_hybatmaerosnowDA_860102e8 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C96_atm3DVar_860102e8/logs/2021122018/gdas_fcst_seg0.log
Error logs:
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_860102e8/logs/2021032418/enkfgdas_fcst_mem001.log
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_860102e8/logs/2021032418/enkfgdas_fcst_mem002.log
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_860102e8/logs/2021032418/gdas_fcst_seg0.log
Error logs:
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C48_S2SWA_gefs_860102e8/logs/2021032312/gefs_fcst_mem001_seg0.log
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C48_S2SWA_gefs_860102e8/logs/2021032312/gefs_fcst_mem002_seg0.log
Error logs:
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C96C48_hybatmDA_860102e8/logs/2021122018/enkfgdas_fcst_mem001.log
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C96C48_hybatmDA_860102e8/logs/2021122018/enkfgdas_fcst_mem002.log
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C96C48_hybatmDA_860102e8/logs/2021122018/gdas_fcst_seg0.log
Experiment C96C48_ufs_hybatmDA_860102e8 Terminated with 0
FAIL
FAIL tasks failed and 3 dead at Wed Jan 22 04:20:59 UTC 2025
Experiment C96C48_ufs_hybatmDA_860102e8 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_860102e8/logs/2021122012/enkfgdas_fcst_mem001.log
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_860102e8/logs/2021122012/enkfgdas_fcst_mem002.log
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_860102e8/logs/2021122012/gdas_fcst_seg0.log
Error logs:
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_860102e8/logs/2024022318/enkfgdas_fcst_mem001.log
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_860102e8/logs/2024022318/enkfgdas_fcst_mem002.log
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_860102e8/logs/2024022318/gdas_fcst_seg0.log
Experiment C48_ATM_860102e8 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Jan 22 05:39:56 UTC 2025
Experiment C48_ATM_860102e8 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C48_ATM_860102e8/logs/2021032318/gfs_metpg2o1.log
Experiment C48_S2SW_860102e8 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Jan 22 07:36:16 UTC 2025
Experiment C48_S2SW_860102e8 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3017/RUNTESTS/COMROOT/C48_S2SW_860102e8/logs/2021032318/gfs_metpg2o1.log

dpsarmie requested review from WalterKolczynski-NOAA, aerorahul and junwang-noaa as code owners October 18, 2024 18:45

dpsarmie marked this pull request as draft October 18, 2024 19:51

junwang-noaa mentioned this pull request Oct 23, 2024

Updating the global tests namelist and configuration file format ufs-community/ufs-weather-model#1663

Open

github-advanced-security bot found potential problems Oct 30, 2024

View reviewed changes

dpsarmie mentioned this pull request Dec 4, 2024

Update global_control.nml.IN based on GW specs ufs-community/ufs-weather-model#2528

Closed

14 tasks

dpsarmie closed this Dec 12, 2024

dpsarmie force-pushed the feature/GW-RT_GlobalNML branch from 51ecf63 to 6c5b9bb Compare December 12, 2024 15:47

dpsarmie and others added 2 commits December 19, 2024 20:32

Finalized parsing changes

8ce541b

Merge branch 'NOAA-EMC:develop' into feature/GW-RT_GlobalNML

e93bf48

dpsarmie reopened this Dec 20, 2024

adjust MOM6 nml changes

eee29fc

WalterKolczynski-NOAA marked this pull request as ready for review December 20, 2024 19:54

WalterKolczynski-NOAA requested a review from jiandewang as a code owner December 20, 2024 19:54

dpsarmie added 2 commits December 27, 2024 13:41

Merge branch 'NOAA-EMC:develop' into feature/GW-RT_GlobalNML

dcd20de

Merge branch 'NOAA-EMC:develop' into feature/GW-RT_GlobalNML

e00afe4

WalterKolczynski-NOAA requested changes Jan 8, 2025

View reviewed changes

dpsarmie and others added 3 commits January 14, 2025 09:36

Merge branch 'NOAA-EMC:develop' into feature/GW-RT_GlobalNML

81a1b4f

Merge branch 'NOAA-EMC:develop' into feature/GW-RT_GlobalNML

c9ac910

Fixes FHZER calls/definitions

f4e23b7

WalterKolczynski-NOAA previously approved these changes Jan 16, 2025

View reviewed changes

Merge branch 'NOAA-EMC:develop' into feature/GW-RT_GlobalNML

85226d5

Fixes typo in PERT_MP

b22742b

Added a missing period

dpsarmie dismissed WalterKolczynski-NOAA’s stale review via b22742b January 16, 2025 14:55

Fix single quotes and PERT_CLDS setting

c4b8b55

Single quotes have been forced on all filenames in the namelist. PERT_CLDS has also been set to .false. unless the DO_SPPT flag is on, then PRET_CLDS is set to .true.

dpsarmie dismissed WalterKolczynski-NOAA’s stale review via c4b8b55 January 21, 2025 20:09

Merge branch 'NOAA-EMC:develop' into feature/GW-RT_GlobalNML

860102e

WalterKolczynski-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Jan 22, 2025

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Jan 22, 2025

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Jan 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update scripts to use the GW-RT global namelist #3017

Update scripts to use the GW-RT global namelist #3017

dpsarmie commented Oct 18, 2024 •

edited

Loading

WalterKolczynski-NOAA commented Oct 24, 2024

dpsarmie commented Oct 25, 2024

dpsarmie commented Oct 28, 2024

dpsarmie Oct 30, 2024

WalterKolczynski-NOAA Jan 8, 2025

WalterKolczynski-NOAA left a comment

WalterKolczynski-NOAA Jan 8, 2025

WalterKolczynski-NOAA commented Jan 16, 2025

dpsarmie commented Jan 16, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

Update scripts to use the GW-RT global namelist #3017

Are you sure you want to change the base?

Update scripts to use the GW-RT global namelist #3017

Conversation

dpsarmie commented Oct 18, 2024 • edited Loading

Description

The following is a Google Doc with new variables that were added to the nml template with this PR: Google Sheets link

Unresolved issues and input needed from GW group before final code review

Type of change

Change characteristics

How has this been tested?

Checklist

WalterKolczynski-NOAA commented Oct 24, 2024

dpsarmie commented Oct 25, 2024

dpsarmie commented Oct 28, 2024

dpsarmie Oct 30, 2024

Choose a reason for hiding this comment

WalterKolczynski-NOAA Jan 8, 2025

Choose a reason for hiding this comment

WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA Jan 8, 2025

Choose a reason for hiding this comment

WalterKolczynski-NOAA commented Jan 16, 2025

dpsarmie commented Jan 16, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

emcbot commented Jan 22, 2025

dpsarmie commented Oct 18, 2024 •

edited

Loading