Test the UFO of SATWND in 3DEnVar

[JingCheng-NOAA]

To document the process of assimilating SATWND data in 3DEnVar.

[JingCheng-NOAA]

First step is creating an IODA nc format observation file from GDAS or GFS bufr file using iodaconv.

A json file is created for the GOES AMV data on Hera "/scratch1/NCEPDEV/hwrf/scrub/Jing.Cheng/jediwork/obs/test/bufr2ioda_satwind_amv_goes.json".
Use script "bufr2ioda_satwnd_amv_goes.py" will create two IODA nc files in out directory:

gfs.t12z.satwnd.abi_goes-16.tm00.nc
gfs.t12z.satwnd.abi_goes-17.tm00.nc

[JingCheng-NOAA]

The YAML file is prepared as /scratch1/NCEPDEV/hwrf/scrub/Jing.Cheng/jediwork/UFOtests/3denvar_jedi_satwnd.yaml.
And the bash script to run the job is /scratch1/NCEPDEV/hwrf/scrub/Jing.Cheng/jediwork/UFOtests/sat-3dvar.sh.
The executable file is changed from fv3jedi_var.x to gdas.x

Currently, the test run encountered error of YAML file parsing issue according to the log.
"Assertion failed: dynamic_cast<const YAMLItemEntry*>(advance) in value, line 362 of /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/cache/build_stage/spack-stage-eckit-1.24.5-uo33x5dafdu4ae5fl6o5ebpdurhmv7af/spack-src/src/eckit/parser/YAMLParser.cc"

[JingCheng-NOAA]

The parsing error is fixed by modifying the background part of the YAML file.
The yaml file is readed in with no issue. However, the test stopped by complaining of missing variable:
23: Field_fail: Field wind_reduction_factor_at_10m cannot be obtained from input fields.

[JingCheng-NOAA]

The parsing error is fixed by modifying the background part of the YAML file. The yaml file is readed in with no issue. However, the test stopped by complaining of missing variable: 23: Field_fail: Field wind_reduction_factor_at_10m cannot be obtained from input fields.

Adding variable 'f10m' in state variable will avoid such error

[JingCheng-NOAA]

Adding filter Background Check will bring up the issue that observations fall out of model domain, and using LAMDomainCheck still have the same error.
Exception: Assertion failed: hofx[jobs] != missing in applyFilter, line 116 of /scratch1/NCEPDEV/hwrf/save/Jing.Cheng/JEDI/GDASApp/bundle/ufo/src/ufo/filters/BackgroundCheck.cc Tried to erase key "mesh[address=0x19e4a70],halo[size=0]" from cache "NodeColumnsHaloExchangeCache" but it was not found. Erased key "mesh[address=0x19e4a70],halo[size=1]" from cache "NodeColumnsHaloExchangeCache".

[JingCheng-NOAA]

For testing purpose only, reprocess the IODA NC file and only select a small chunk of observation within HAFS domain to test if the experiment can run through the Background Check.
Original observations

Reduced Observations

The analysis using Background Check is able to completed with no error with this small chunk of observations

[yonghuiweng]

Here is the plot when you cut off 5deg around the boundary.

We can see some points around the bottom sides are still out of the domain.
Here is the domain and obs by cut off 10 deg.

[JingCheng-NOAA]

@yonghuiweng Thanks for the plots. It explains!

[JingCheng-NOAA]

OBSValue WindEsatward

OBSValue WindNorthward

OMB WindEastward

OMB WindNorthward

[JingCheng-NOAA]

In HAFS GSI, the assimilated SATWIND ObsTypes are all from GOES satllites: 240, 241, 245, 246, 247, 251.
Besides type 241, all the types are tested in JEDI. The observation Operator YAML is recorded.
ObsType 241 are GOES CIMSS AMV, currently only available after year 2023, which is beyond the testing window (20200825).
There is no currently available json/python for converting this types of AMV into NC file, and no available of Observation Operator as well. This type of data will need revisit in the future.

[JingCheng-NOAA]

Continue the test with updated SATWND yaml file with or without domain filter. The QCflag did show adding the domain filter, removing observations due to out of domain. However, it seems like another process of QC (metadata too farm from reference) is playing an role too, thus the final observation number that passed the QC process are the same.
Without online domain filter

passed QC: 19143
monitored(calculated hx but not assimilated): 0
reject flag 10: 0
reject flag 11: 0
reject flag 12: 356907
reject flag 13: 0
reject flag 14: 552636
reject flag 15: 0
reject flag 16: 0
reject flag 17: 25360

With online domain filter

passed QC: 19143
monitored(calculated hx but not assimilated): 0
reject flag 10: 0
reject flag 11: 0
reject flag 12: 809646
reject flag 13: 0
reject flag 14: 125253
reject flag 15: 0
reject flag 16: 0
reject flag 17: 4

A plot show that even without online domain filter, the JEDI still can remove the data out of domain boundary.