RT coastal_irene_atm2roms is not reproducing on Frontera

[uturuncoglu]

As a part of the work that aims to bring support for Frontera in UFS Coastal, i created baseline also for ROMS configuration but it is failing in baseline check as following,

==> logs/log_frontera/rt_005_coastal_irene_atm2roms_intel.log <==

baseline dir = /work2/01118/tg803972/frontera/RT/NEMSfv3gfs/develop-20240126/coastal_irene_atm2roms_intel
working dir  = /scratch1/01118/tg803972/frontera/tg803972/FV3_RT/rt_210316/coastal_irene_atm2roms_intel
Checking test 005 coastal_irene_atm2roms_intel results ....
 Comparing irene_avg.nc .........NOT OK
 Comparing irene_his.nc .........NOT OK
 Comparing irene_rst.nc .........NOT OK

The total amount of wall time                        = 271.812085
The maximum resident set size (KB)                   = 320540

Test 005 coastal_irene_atm2roms_intel FAIL

This tests is reproducing on other platforms like Orion and Hercules. It might be related with the used compiler on Frontera. At this point, I am trying to install NCAR's cprnc tool to Frontera to check the files.

[uturuncoglu]

This is also case for Hercules Intel. The NCAR cprnc tool shows that the data are identical with baseline but UFS uses nccmp command and fails. So, I'll look at it further.

[uturuncoglu]

It could be a bug in baseline checking - ufs-community/ufs-weather-model#2245

[uturuncoglu]

The Hercules issue was related with the global attributes since it has compiler flags. I am plaining to add export CMP_DATAONLY=true to ROMS test since we would not want to create the baseline every time once global attributes are changed. I'll check this on Frontera again and check this fix the Frontera issue too.

[uturuncoglu]

@janahaddad @pvelissariou1 Okay. I pushed coupled of fixed to RT system related to the Frontera and now ROMS case pass the baseline. Closing this.