Advanced Topics

Debug mode

To run the UFS-S2S-model in debug mode, you must compile each component in debug mode and use the ESMF debug module.

Turning on debug in compilation

An update will be coming to allow the setting of a single DEBUG=Y flag at the top of the build for the UFS-S2S-model. Until then, each component must be set manually to DEBUG mode.

For NEMS and FV3:
In conf/configure.fv3.{machine} in the FLAGS section set: DEBUG = Y

For CICE:
In CICE/bld/Macros.Linux.NEMS.$MACHINE_ID set: DEBUG = Y

For MOM6:
In MOM6/compile.sh, set: 
    #to build debug mode
    REPRO=
    DEBUG=1
    #otherwise
    #REPRO=1
    #DEBUG=

Note: The debug settings have been added to the appropriate files for machines Hera and Cheyenne. Developers working on other platforms can use these as templates for porting the debug settings to other platforms. If a new platform has debug support added, please create an issue and an accompanying PR in the UFS-S2S-model repository so that these can be made available to the wider community.

Using ESMF in debug mode

To do this you need to change the loaded ESMF module. The module file location can be found by looking in the conf/before_components.mk file at the CHOSEN_MODULE variable. For computing system Hera, the location is modulefiles/hera.intel/fv3_coupled. To use ESMF in debug mode, change

module load esmf/8.0.0

to

module load esmf/8.0.0g

Running UFS-S2S-model in debug mode

Some things to keep in mind when running in debug mode:

1. The default walltime settings will not be long enough. To change the walltime settings in the C384 cold start test for example, add the setting

walltime=10800   # seconds

to the section defining the cold c384 test in fv3mom6cice5.input. On Hera, in debug mode for the C384 case, the cold start requires approximately 1 hour of walltime.

2. The regression tests will fail since they have been generated with the model compiled in non-debug mode.

Restarting the coupled model

This section is under construction.

The list of restart files produced by the UFS-S2S-model can be found here

Changing the number of PEs for FV3

The following variables all need to be changed:

In input.nml: layout and ntiles

In model_configure: TASKS, quilting, write_groups, and write_tasks_per_group

The definitions of variables and file where they are located:

TASKS = total number of all tasks for all components (model_configure)
layout = INPES, JNPES is the layout of pets on each task in the x & y directions (input.nml)
ntiles is the number of tiles, typically 6 (input.nml)
quilting true/false variable to use writer cores for FV3GFS (model_configure)
write_groups is the number of write groups for FV3GFS (model_configure)
write_tasks_per_group is the number of tasks per FV3GFS write group (model_configure)

The number of FV3 tasks is then given by:

(INPES x JNPES x 6) + (write_groups x write_tasks_per_group)

The PET layout for each component then needs to be adjusted consistent with the TASKS. The default values are set in compsets/MACHINEID.input (e.g. default_cpl in compsets/hera.input). If the number of PEs for FV3 is changed, non-default values consistent with the total number of tasks will need to be added to the appropriate compset test. Copy the values from default_cpl and insert into the compset, changing values where needed to be consistent with the number of FV3 tasks.

Miscellaneous

Turning off ESMF PET logs

In parm/model_configure.IN, ESMF PET logs are turned on automatically. To turn off ESMF PET logs, change

print_esmf: .true.

to

print_esmf: .false.

Creating instantaneous CICE5 history files

By default, the CICE5 history files are 6 hourly means. For comparison to FV3 output, it is often helpful to have CICE5 history files as instantaneous 6 hour snapshots. To turn off averaging in the CICE5 output, set

 CICE_HIST_AVG='.false.'

in the appropriate compset test.