Port rrfs-workflow to Hercules

[guoqing-noaa]

DESCRIPTION OF CHANGES:

• Port rrfs-workflow to Hercules

TESTS CONDUCTED:

• On machines/platforms:

• WCOSS2

• Hera

• Orion

• Hercules

• Jet

• Test cases:

• Non-DA engineering test
• DA engineering test
• Other sample scripts:
  ush/sample_configs/others/config.sh.RRFS_CONUS_3km. Period: 1 February 2022.

ISSUE:

• Fixes the issue(s) mentioned in Port rrfs-workflow to Hercules #171

CONTRIBUTORS (optional):

@chan-hoo @ShawnMurdzek-NOAA @JacobCarley-NOAA

[chan-hoo]

@guoqing-noaa, thanks for this PR. I got the following error:

/usr/bin/env: ‘python’: No such file or directory

This is because python does not exist in /usr/bin/env. You can find a temporary solution for this here (aqm_dev):
ufs-community/ufs-srweather-app@6884fb0
I added a new file "checkout_externals_python3". If you have any better solution, you can do it :)
Hope someone opened a ticket for this to the hercules help desk.

[JacobCarley-NOAA]

I believe @JamesAbeles-NOAA is also working on a Hercules port. I am including him here for awareness.

[JacobCarley-NOAA]

One more thing, the GSI Utils folks are also working on porting to Hercules and reported an MPI environment setting that is needed to be enabled to get things to run properly with netCDF4: export I_MPI_EXTRA_FILESYSTEM=ON. This was also reported by the ufs-weather-model folks. I wanted to bring this up here in case anyone was not aware.

[guoqing-noaa]

@chan-hoo Right, this is because Hercules does not have a default Python env. Users can module load python before checking out sorc and/or do app_build.sh.

One way to solve this is to describe this in README.md.
Another way is to add one line in app_build.sh before checkout_externals:

python --version
if [[ $? -ne 0 ]]; then
   module load python
fi

The caveat of the second method is that it is only needed in Hercules.

Let me know which method you guys prefer. Thanks!

[guoqing-noaa]

One more thing, the GSI Utils folks are also working on porting to Hercules and reported an MPI environment setting that is needed to be enabled to get things to run properly with netCDF4: export I_MPI_EXTRA_FILESYSTEM=ON. This was also reported by the ufs-weather-model folks. I wanted to bring this up here in case anyone was not aware.

Very important information! Thanks, @JacobCarley-NOAA! I have not run the DA engineering test yet. I will do it right now.

[guoqing-noaa]

Another thing I'd like to mention is that I did not find the "yafyaml" library on Hercules. Currently, I commented it out in the Lua file. Do any scripts in rrfs-workflow use "yafyaml"?

[chan-hoo]

@chan-hoo Right, this is because Hercules does not have a default Python env. Users can module load python before checking out sorc and/or do app_build.sh.

One way to solve this is to describe this in README.md. Another way is to add one line in app_build.sh before checkout_externals:

python --version
if [[ $? -ne 0 ]]; then
   module load python
fi

The caveat of the second method is that it is only needed in Hercules.

Let me know which method you guys prefer. Thanks!

Vote for the second one (app_build.sh) :)

[guoqing-noaa]

Vote for the second one (app_build.sh) :)

Good. I will make this change as voted, :)

[chan-hoo]

@guoqing-noaa, I got

 python: command not found

[guoqing-noaa]

@guoqing-noaa, I got

 python: command not found

@chan-hoo This is expected on Hercules. Do you mean I need to suppress this error message? such as:
python --version 1>/dev/null 2>/dev/null

This error message will NOT stop the checkout process. However, Hercules has a performance issue today and now I cannot even do "ls". So it may look like "python not found" hangs the whole process just by reading the output message.

[chan-hoo]

Got it. I'll try it again tomorrow.

[guoqing-noaa]

@chan-hoo FYI, Hercules is back.

[chan-hoo]

@guoqing-noaa, I am testing it now. So far so good. I'll let you know once it is done.

[chan-hoo]

@guoqing-noaa, test was complete successfully. Approving. Happy Holidays! :)

[MatthewPyle-NOAA]

Thanks for testing and approving, @chan-hoo

[guoqing-noaa]

Thanks, @chan-hoo @MatthewPyle-NOAA! Happy holidays!