TEST IASI METOP-B in HAFS domain

[XuLu-NOAA]

This is to document the test of assimilating IASI Metop-B radiance data.
Work will include the following steps:

1. Convert budr data to ioda with bufr2ioda tools
2. Convert bias file to ioda with bufr2ioda tools
3. Modify the yaml file from ATMS test accordingly
4. Check the assimilation results
5. Add into the test case.

[XuLu-NOAA]

Initial test starts from modifying the yaml template from parm/jcb-gdas/observations/atmosphere/iasi_metop-b.yaml.j2.
Single ob and multiple obs tests indicate similarities between GSI and JEDI. Major differences may come from CRTM version 2.4.0.1 in GSI versus version 2.4.1 in JEDI.
https://docs.google.com/presentation/d/13asklpaQm2xLYRJp7aPwPQOPVbQ9SAbuJ0jk3G4PRMY/edit#slide=id.g2fc6f3057c3_1_20

Note, by default, the bufr data is radiance, so the yaml file will convert radiance to brightness temperature.
In the current tests, the data is from GSI diagfile, so it is already brightness temperature. So I commented out the conversion inside JEDI yaml. Needs to be further tested from the bufr file conversion.

[XuLu-NOAA]

Initial trial using the GDASApp yaml to convert radiance into BT failed in my test run. Error messages show:
Group name for observed values is neither ObsValue nor DerivedObsValue
Line by line comparison with the working yaml from Emily did not find issues, but still not work.
Failed exp are logged here:
/work/noaa/hwrf/scrub/xulu/JEDI_Dual_new/2024063012/02L/diff_test
Will keep working with Emily to figure out the issue.

[XuLu-NOAA]

Preliminary investigation with Emily did not find the issue in the yaml files.
@emilyhcliu will test on global GDASApp and see if similar error occurs, and debug from there.

[XuLu-NOAA]

New findings indicate the issue happens when the derived BT does not fall into the regional domain. This might be due to the fact that the code is originally designed for global domain.
Working on get-around to skip those abortion checks.

[XuLu-NOAA]

The issue was found due to
https://github.com/JCSDA-internal/ufo/blob/6b56db2fbb36d5e6959ca0d2de701911734c27f4/src/ufo/filters/VariableTransforms.cc#L69

When there is no obs falling in the regional DA domain, it will skip the rest of the transform procedures. Even though we don't have the obs to transform from, the rest of this subroutine appears to be still necessary to initialize certain variables etc...

If I comment out this return line. The whole code will go through with 0 obs available.
An UFO issue https://github.com/JCSDA-internal/ufo/issues/3580 is created for further discussions.