Adding new pod: KeyError pod name not found

[bitterbark]

I'm adding a brand new POD for the first time and must be missing a step. The error I'm getting is:
File "/home/bundy/mdtf/MDTF-diagnostics/src/diagnostic.py", line 493, in from_config
return cls.from_struct(pod_name, config.pod_data[pod_name])
KeyError: 'blocking_neale'

The driver script is diagnostics/blocking_neale/blocking_neale.py
I am calling the POD from an input jsonc file
"pod_list": [
"blocking_neale"
]

The framework runs just fine with other (existing) pods.

In debugging, I have found a list of (some) existing pods in install.py and added the new one there:
if d.env_setup == 'conda-basic':
d.pods = ["Wheeler_Kiladis", "EOF_500hPa", "MJO_suite", "MJO_teleconnection","blocking_rneale"]
but that didn't help.

Is there somewhere else with a master list of acceptable pods? Or what else am I missing?

[bitterbark]

There seems to be a related fix brought up in issue #156

I'm not sure if this will fix my problem though, because my settings file passes
the online JSONC validator (BTW the other validator mentioned in responses in #156 fails with the in-file comments) .

[tsjackson-noaa]

Hi Dani,

• install.py is the 'all-in-one' installer script that was used for CI at one point, but isn't currently used.
• You may have a typo in the list of requested PODs (blocking_neale vs. blocking_rneale above).
• See PR Improve error handling for POD mis-specification #183, which improves the error messages in this case.

The criteria for a POD to be recognized by the framework are 1) a subdirectory of /diagnostics with the POD's name, and 2) a well-formed settings.jsonc file inside that directory. Could you reproduce the contents of that file here?

[bitterbark]

// Example MDTF diagnostic settings file
//
// This should be an ascii text file in JSON format
// (https://en.wikipedia.org/wiki/JSON#Data_types_and_syntax), with the addition
// that all text to the right of a "//" is a comment and ignored (sometimes
// called "JSONC" format).
//
{
"settings" : {
// Name of the driver script. This should be a single script, taking no
// command-line arguments, which runs the entire analysis (calling any other
// scripts or subroutines as needed.)
"driver" : "blocking_neale.py",

// Human-readable name of the diagnostic. May contain spaces. 
"long_name" : "Rich Neale's blocking diagnostic",

// Modeling realm. If your diagnostic uses data from multiple realms, give
// this as a list.
"realm" : "atmos",

// Human-readable name of the diagnostic. May contain spaces. This
// is used to describe your diagnostic on the top-level index.html page.
"description" : "Rich Neale's blocking diagnostic",

// (optional) Any diagnostic-specific environment variables, given as 
// key:value pairs. The user can't change these at runtime, but they can be 
// used to set site-specific installation settings for your diagnostic.
"pod_env_vars" : {
  "EXAMPLE_FAV_COLOR": "blue"
},

// Programs and third-party libraries your diagnostic needs to run. This is
// a set of key:value pairs, where "key" is the name of a language's 
// executable and "value" is a list of all the libraries that language needs.
// The framework will ensure everything listed here is on your diagnostic's
// $PATH when it runs.
"runtime_requirements": {
  "python3": [],
  "ncl": ["contributed", "gsn_code", "gsn_csm"]
}

},

// In this section, you define the names and units for the coordinates used by
// the variables defined below in the "varlist" section. Names in the
// "dimensions" attribute for each variable must correspond to a coordinate
// named here.
"dimensions": {
"lat": {"standard_name": "latitude"},
"lon": {"standard_name": "longitude"},
"time": {"standard_name": "time"}
},

// In this section, you describe what model data your diagnostic uses. Add one
// list entry (the section in curly braces) for each variable, separated by
// commas.
"varlist" : {
// Variable identifier. An environment variable with this name will be set
// to what the corresponding field is called in the model being analyzed.
"zg": {
// Standard name for the variable according to the CF conventions. This
// is used to reliably identify the variable if it's called by a different
// name in the output format of the model being analyzed.
"standard_name" : "geopotential_height",

  // Units the script expects this variable to be in (UDUnits2- compatible
  // string). If the model being analyzed provides the variable in different
  // units, the MDTF package will do unit conversion before calling the script.
  "units": "m",

  // Time frequency the data should be sampled at. Currently recognized
  // values are '1hr', '3hr', '6hr', 'day' and 'mon'.
  "frequency" : "day",

  // Coordinates of the variable (defined in the section above.)
  "dimensions": ["time", "lat", "lon"]
}
// If we needed a second variable, we'd list it here.

}

[bitterbark]

That's the settings file. It passes the online checker
Good eye on the neale vs rneale but that didn't fix it (as you said, that install file isn't used anymore).

% grep blocking src/input_dani.jsonc diagnostics/blocking_neale/blocking_neale.py diagnostics/blocking_neale/settings.jsonc
src/input_dani.jsonc: "blocking_neale"
diagnostics/blocking_neale/blocking_neale.py:print("blocking_neale.py calling blocking.ncl")
diagnostics/blocking_neale/blocking_neale.py:print("blocking_neale.py finished.")
diagnostics/blocking_neale/settings.jsonc: "driver" : "blocking_neale.py",

% grep blocking src/input_dani.jsonc diagnostics/blocking_neale/blocking_neale.py diagnostics/blocking_neale/settings.jsonc | grep -i rneale

[bitterbark]

BTW, How do I tell when the code from a pull request is on the develop branch? This one says approved by reveiews as of this afternoon but the latest on develop is one from @wrongkindofdoctor yesterday...

[bitterbark]

Solved: the culprit was the hard-coded path in mdtf executable. My work flow doesn't trust git as much as it should, so I had made a copy as my sandbox, which had the updated input.jsonc, but was still pointing to the old path. If there was a way to set it relative that would be preferable for me, but I can imagine there are ramifications I haven't considered.
ie.
OLD _mdtf="/home/bundy/mdtf/MDTF-diagnostics-blocking"
NEW _mdtf="."

Anyway, now I can deal with actual errors of my implementation.

[tsjackson-noaa]

Hi @bitterbark,
Glad you found the source of the problem! I set up the wrapper script with an absolute path in case the user calls the wrapper script when their CWD is somewhere other than the directory the script is in.

The contents of pull requests need to be merged manually into /develop, or whatever the target branch is -- it's not done automatically once the code review is done. If you look at the closed PRs, GitHub color-codes the ones that have been merged into their target as purple, while those that were closed without merging are red. The pull request that's being merged is also referenced in the commit message on the /develop branch for that merge (e.g. 6202072).

[tsjackson-noaa]

Marking as 'answered' because @bitterbark found the cause of the problem in this reply. GitHub doesn't allow replies to comments to be marked as answers.