Debug Online Domain Check with ATMS radiance DA

[XuLu-NOAA]

This is to document the exploration of online domain check for the HAFS-JEDI workflow.
Based on https://github.com/JCSDA-internal/mpas-jedi/pull/976, an online domain check is available.
However, early investigations showed this approach works fine for soundings and satwnd but not for radiances like ATMS in HAFS-JEDI.
Further investigations showed that the failure happened to a few data points near the domain boundary where the land/sea fractions are not summed to 1. It appears the interpolation near the boundary has potential issues and the 0 values of the out-of-domain grids are involved.

An easy temporary fix is proposed in https://github.com/JCSDA-internal/ufo/pull/3554 to skip those issue points.
Discussions are continued in this UFO PR to further resolve the boundary interpolation bugs.

[XuLu-NOAA]

Additional suggestions from Hui that the "reduce obs space" configuration would work with the online domain check and help reduce memory burdens by pre filter the obs before the check. Examples from CONUS domain are:

         - filter: Domain Check
           apply at iterations: 0,1
           where:
             - variable:
                 name: MetaData/latitude
               minvalue: 50.0
               maxvalue: 23.0
           action: 
              name: reduce obs space

         - filter: Domain Check
           apply at iterations: 0,1
           where:
             - variable:
                 name: MetaData/longitude
               minvalue: -63
               maxvalue: -129
           action: 
              name: reduce obs space

[XuLu-NOAA]

Updates on the reduce obs space, the runtime reduced from 492.79s to 224.95s. The hofx file also contains less raw data:
Without reduced obs space:

With reduced obs space:

The two green stars marks the location of sum issue with land/sea fractions

[XuLu-NOAA]

Updates:
Running the same fix in another DA cycle experienced a different error:
214: Exception: Assertion failed: hofx[jobs] != missing in applyFilter, line 116 of /work/noaa/hwrf/save/xulu/JEDI/newest/GDASApp20241209/bundle/ufo/src/ufo/filters/BackgroundCheck.cc
This is still caused by the data points outside the domain or too near to the boundary.
A temporary fix to skip those missing values in BackgroundCheck.cc can work. But still needs to solve the root of the boundary interpolation following the discussion in the ufo PR3554.

[XuLu-NOAA]

Updates:
The domain check and all other errors are now gone with the fixes in the FV3-JEDI halos in the PR3554 discussion.