Incorporating JEDI into HAFS workflow

[XuLu-NOAA]

This is to document the effort of bringing JEDI into the HAFS workflow based on the https://github.com/hafs-community/HAFS/tree/feature/hafs_jedi branch.

Currently, the changes are not pushed to the branch and are stored locally on Hera as we are still optimizing the workflow:
/scratch1/NCEPDEV/hwrf/save/Xu.Lu/hafsv2_jedi

Implemented Capabilities:
Single-resolution Configurations work fine now with the cycling of
3DEnVar, 4DEnVar, 3DFGAT

Capabilities to be developed:
Dual-resolution DA capability
#17

To be optimized issues in the workflow:

1. The rocoto job atm_prep_ens currently depends on RUN_ANALYSIS==YES, it works fine for JEDI DA or GSI with self-cycled HAFS ensemble. But for situations where RUN_ANALYSIS==YES and RUN_ENSDA==NO, it is redundant.
   Still figuring out a better way to work around the or condition in rocoto xml.
2. In jobs/JHAFS_ANALYSIS, lots of JEDI environment conditions need to be taken care of in ${USHhafs}/hafs_pre_job.sh.inc
3. To be added...

[XuLu-NOAA]

Moving all the work on Orion since the python module on Hera does not work with the offline domain filter scripts.
The new workflow location is:
/work/noaa/hwrf/save/xulu/hafs_jedidual

Dual-Resolution DA capability has been added and tested.
The workflow now can work with ATMS, AMSUA, Satwnd and Sondes.
But there are currently two issues:

1. runtime and memory increases with the number of cores requested, making the DA several hours long with 40 member and full dataset. Working with JCSDA on the issue.
2. The analysis wind fields in JEDI is ua/va on A-grid, needs to convert them to u/v on D-grid for HAFS initialization. Working with Yonghui to develop the capability in HAFS DA tools.

[XuLu-NOAA]

Using hafstool to interpolate the ensemble onto DA grid prior to JEDI reduces the runtime and memory cost.
Now the JEDI-HAFS workflow is almost working properly with the online domain check and newest version of JEDI.
The issues left are:

1. There are boundary interpolation issues for all kinds of obs. Temporary fixes are patched to get around those problematic data points due to missing values. But still need to wait a fix in the interpolation from https://github.com/JCSDA-internal/ufo/pull/3554
2. The IASI assimilation is still not working properly in the radiance to BT conversion. Need to be solved by TEST IASI METOP-B in HAFS domain #21
3. Working with Yonghui on interpolating the A-grid wind back onto D-grid.