Enhancement: Update pb2nc to work with bufrlib 12.1.0

[georgemccabe]

While testing the MET conda recipe for dtcenter/METplus#2863, I discovered that pb2nc crashes when using bufrlib 12.1.0, which is the version that is provided via NCEPLIBS-bufr in conda-forge. This version of bufrlib also supports the Intel OpenAPI compilers (see release notes).

The following error is output:

 ***********BUFR ARCHIVE LIBRARY ABORT**************
 BUFRLIB: PARUSR - INPUT STRING () HAS

                   NO STORE NODES

 ***********BUFR ARCHIVE LIBRARY ABORT**************

Describe the Enhancement

Update calls to bufrlib for pb2nc to prevent this error.

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Review default alert labels
• Select component(s)
• Select priority
• Select requestor(s)

Milestone and Projects

• Select Milestone as a MET-X.Y.Z version, Consider for Next Release, or Backlog of Development Ideas
• For a MET-X.Y.Z version, select the MET-X.Y.Z Development project

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issue
  Select: Milestone as the next official version
  Select: MET-X.Y.Z Development project for development toward the next official release
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[jprestop]

This error exists when compiling bufrlib version 12.1.0 and MET version 12.0.0 using the Intel oneAPI compilers version 2024.2. Looking at NCEPLIBS-bufr Issue NOAA-EMC/NCEPLIBS-bufr#538, I see that support for the oneAPI compilers was officially added with bufrlib version 12.1.0. Looking at the PR Add IntelLLVM support #536, I see the following "conflicts("%oneapi@:2024.1", msg="Requires oneapi 2024.2 or later")", so it looks like bufrlib version 12.1.0 requires oneAPI 2024.2 or later, which is being used for this compilation. I have posted an issue in the NCEPLIBS-bufr that describes the problem (NOAA-EMC/NCEPLIBS-bufr#640).

[JohnHalleyGotway]

Met with @georgemccabe and @jprestop about this issue on Jan 29, 2025.

This runtime error occurs only when running certain configurations of pb2nc, notably when obs_bufr_var is set to an empty list, as described in this issue comment.

We also need to check about successful derivation of observations (e.g. RH, DPT, etc), as this might be related.

Issue occurs with:

• Intel compilations on seneca and other HPC.
• GNU compilations on seneca installed via a conda recipe - difficult to debug in this environment.

It does not occur with:

• GNU compilations directly on seneca.
• GNU compilations through DockerHub and used in GitHub actions.
• Clang compilations directly on Mac laptop, although derivation issue still exists.

Recommend debugging this behavior with Intel builds on:

1. As the met_test user in seneca:/d1/projects/MET/intel.
2. As the dtcrt user in casper:/glade/work/dtcrt/METplus/casper/components/MET/development/12.1.0/intel_oneapi.

This issue originally arose via the USAF optimization project but also came up during the NCAR-Base conda install work.

Next steps:

• @georgemccabe will email NRIT ([email protected]) to ask for advice about an existing or new degugger for the Intel oneapi compiler on seneca.
• @georgemccabe will email CISL ([email protected]) to ask for advice about an existing or new debugger for the Intel oneapi compilers on derecho and casper.
• @JohnHalleyGotway will test the seneca build to confirm that emptying out the obs_bufr_var array really does cause the error to occur with a previously working configuration.

[hsoh-u]

-index option provides the available variable list.

 pb2nc
        prepbufr_file
        netcdf_file
        config_file
        [-index]

 "-index" indicates that the meta data (available variables and headers) is extracted from "prepbufr_file" (optional). No NetCDF outputs. "-obs_var" is ignored.

[JohnHalleyGotway]

@georgemccabe and @JohnHalleyGotway confirmed that the PARUSR problem occurs when pb2nc is compiled using the -O2 optimization flag but does not occur when using the -g debugging flag without -O2. When -g -O2 are both used, it occurs.

For the -O2 failure, I changed obs_bufr_var from an empty list to a populated one and retested:

< obs_bufr_var = [ ];
> obs_bufr_var = [ "QFCMOD", "TFCMOD", "VFCMOD", "UFCMOD", "ZFCMOD", "PFCMOD", "WVWTA", "WVWTG", "QVWTA", "QVWTG", "TVWTA", "TVWTG", "PVWTA", "PVWTG", "HBLCS", "CLTP", "CLAM", "QOETU", "QCS", "QCL", "QAN", "QOE", "QFC", "QRC", "QPC", "QQM", "ESBAK", "QOB", "TOETU", "TCS", "TCL", "TAN", "TOE", "TFC", "TRC", "TPC", "TQM", "TOB", "TDO", "TVO", "WOETU", "FFO", "VCS", "UCS", "VCL", "UCL", "VAN", "UAN", "WOE", "VFC", "UFC", "WRC", "WPC", "WQM", "RF10M", "DFR", "DFP", "DFQ", "VOB", "UOB", "DDO", "ZCS", "ZCL", "ZAN", "ZFC", "ZRC", "ZPC", "ZQM", "ZOB", "CAT", "POETU", "PCS", "PCL", "PAN", "POE", "PFC", "PRC", "PPC", "PQM", "POB", "XDR", "YDR", "HRDR", "PROCN", "SQN", "TCOR", "RPT", "SIRC", "SSTQM", "SST1", "TSB", "RCT", "DGOT", "HTOI", "HBOI", "AFIC", "HTOT", "HBOT", "HOCB", "RFFL", "QIFN", "QIFY", "TOCC", "PSW2", "PSW1", "PRWE", "HOVI", "PMQ", "PMO", "CHPT", "3HPC", "TP01", "MXGS", "VSSO", "TOSD", "TP06", "TP03", "DOFS", "MITM", "MXTM", "24PC", "TP12", "TP24", "ALSE", "SQM", "SOB", "VTVI", "PKWDSP", "PKWDDR", "HOWV", "POWV", "TOPC", "HOCT", "MSST", "HOWW", "POWW", "ASMP", "TDMP", "DOSW", "HOSW", "POSW", "XS20", "XS10", "PWE", "PWR", "PWP", "PWQ", "PWO", "PRSS", "EXPRSRD", "RSRD", "BSCD", "WVCQ", "LKCS", "CTCN" ];

And it ran WITHOUT error.

So the problem exists when running pb2nc compiled with -O2 and with an empty list for obs_bufr_var = [ ];.

I note that the BUFRLIB to which MET links is also compiled with -O2 in:

seneca:/d1/projects/MET/intel/external_libs/bufrlib/NCEPLIBS-bufr-12.1.0/build/CMakeFiles/CMakeOutput.log

So the problem occurs when pb2nc and BUFRLIB are both compiled using -O2.