How to present static files in the catalogs?

[aradhakrishnanGFDL]

The catalog builder developed by GFDL is looking into cataloging the static files (e.g. ocean.static.nc). I imagine one of the following ways to do this

1)For every static file, we could get all the variable names in the dataset.
Then the csv catalog will have as many entries based on the variable_id , pointing to the same ocean.static.nc path.
e.g.
exp, model, ..., xt_ocean,..., ocean.static.nc
exp, model, ..., yt_ocean,..., ocean.static.nc
..

2. We can have one entry in the catalog to record the path of the static dataset, and call variable_id "static" (or whatever is more standardized). In this case, the user with the POD can get to the static dataset through intake-esm, by searching based on experiment, realm, model, but then the user needs to use XR or something else to inquire the actual variables in the static file.

[1] requires opening the netcdf file (one per dataset) (this will nicely fit into the CMIP vocabulary so we can "fx" as the miptable and provide distinct variable names. Noting that the clubbing of all the static variables in one netcdf is not the cmip way, yet there needs to be some support to handle it in the catalog builder without compensating the performance )

[2] can handle a lightweight catalog builder.
[3] could be something else

What is MDTF's requirement for static files presented in the catalog ?

thoughts? ideas?

[wrongkindofdoctor]

@aradhakrishnanGFDL The catalog should have fx in the frequency column as shown for the land area fraction data (sftlf; CMIP standard)](https://github.com/wrongkindofdoctor/MDTF-diagnostics/blob/debug_wwes/diagnostics/WWEs/esm_catalog_CESM2_tauu_historical_r1i1p1f1.csv) in the incoming WWEs POD. The settings file varlist entries for static variables do not need a frequency attribute. Instead, an is_static attribute in the varlistentry object populated by the framwork from the setttings file is set to true for time-invariant data, and the framework catalog query will automatically search for frequency = 'fx`.

[aradhakrishnanGFDL]

@wrongkindofdoctor Using fx sounds great!
table_id = 'fx' and frequency = 'fx' is consistent with CMIP vocabulary.

The GFDL PP datasets are organized in the PP components directory, which sadly indicates frequency also in the names, which I will ignore for this context.

Let's see if this is sufficient to describe both CMIP and GFDL statics:
All the following options are demonstrated in [this notebook] (https://github.com/aradhakrishnanGFDL/canopy-cats/blob/608df3f824ed77bf432118dcad4c83023ba19d68/notebooks/statics.ipynb). If you want to look at the data catalogs, please follow the paths in the notebook, assuming you have access to GFDL resources. If you are an outside collaborate and would like to see the catalogs pertaining to these notebooks, send a request to @aradhakrishnanGFDL or open a GitHub issue in canopy-cats repo where the notebook resides.

CSV headers: activity_id,institution_id,source_id,experiment_id,frequency,realm,table_id,member_id,grid_label,variable_id,time_range,chunk_freq,platform,dimensions,cell_methods,standard_name,path

For all the following, the catalog values will have "fx" as table_id and frequency. 
realm will be different. 
time_range, chunk_freq will be empty or  na. Should work? The GFDL builder sets "ts" to cell_methods ATT, which we need to revise later. Separate discussion.


/archive/John.Krasting/fre/FMS2024.02_OM5_20240724/CM4.5v01_om5b06_piC_noBLING/gfdl.ncrc5-intel23-prod-openmp/pp/atmos_month_aer/[atmos_month_aer.static.nc](http://atmos_month_aer.static.nc/)

/archive/John.Krasting/fre/FMS2024.02_OM5_20240724/CM4.5v01_om5b06_piC_noBLING/gfdl.ncrc5-intel23-prod-openmp/pp/ocean_monthly/[ocean_monthly.static.nc](http://ocean_monthly.static.nc/)

/archive/John.Krasting/fre/FMS2024.02_OM5_20240724/CM4.5v01_om5b06_piC_noBLING/gfdl.ncrc5-intel23-prod-openmp/pp/atmos_cmip/[atmos_cmip.static.nc](http://atmos_cmip.static.nc/)

BUT: Multiple variables in one static dataset:

Option 1: intake-esm multiple-variables support:

https://intake-esm.readthedocs.io/en/stable/how-to/use-catalogs-with-assets-containing-multiple-variables.html).

Unlike CMIP, there are MULTIPLE static variables in a single static dataset (below).

Pros: in-built feature in intake-esm. Even if there 10 static variables in a GFDL static dataset, the returned xarray object only has what we searched for. Efficient.

Cons: Though there is an intake-esm convention-ish to pass a list of variable_id(s) to indicate for a single asset,  we are unable to leverage that because then we cannot provide the matching standard names for each of the static variables, that MDTF needs. Catalog builder needs to "open" datasets to query for all variable names, slower runtime.

oo..does not look it's interoperable with other single-variable datasets in the same catalog.

Q: Is this interoperable with the json aggregation we currently have? To be explored. Notebook example in the works.

Option 2:

Catalog each variable in the static dataset with the same path like any other variable.
Pros: Users can "search" for the static variables like any other field, so the flow is uninterrupted.
Cons: the xarray object will contain ALL the variables even though variable_id says "areacello" for instance. So, searching each of the variables and loading them results in duplicates, though it works, it is not efficient.
Catalog builder needs to "open" datasets to query for all variable_ids, slower runtime.

activity_id,institution_id,source_id,experiment_id,frequency,realm,table_id,member_id,grid_label,variable_id,time_range,chunk_freq,platform,dimensions,cell_methods,standard_name,path
dev,,fre,CM4.5v01_om5b06_piC_noBLING,mon,ocean_monthly,fx,,,areacello,,,gfdl.ncrc5-intel23-prod-openmp,,ts,'Ocean Grid-Cell Area',/archive/John.Krasting/fre/FMS2024.02_OM5_20240724/CM4.5v01_om5b06_piC_noBLING/gfdl.ncrc5-intel23-prod-openmp/pp/ocean_monthly/[ocean_monthly.static.nc](http://ocean_monthly.static.nc/)
dev,,fre,CM4.5v01_om5b06_piC_noBLING,mon,ocean_monthly,fx,,,sftof,,,gfdl.ncrc5-intel23-prod-openmp,,ts,'Sea Area Fraction',/archive/John.Krasting/fre/FMS2024.02_OM5_20240724/CM4.5v01_om5b06_piC_noBLING/gfdl.ncrc5-intel23-prod-openmp/pp/ocean_monthly/[ocean_monthly.static.nc](http://ocean_monthly.static.nc/)
dev,,fre,CM4.5v01_om5b06_piC_noBLING,mon,ocean_monthly,fx,,,hfgeou,,,gfdl.ncrc5-intel23-prod-openmp,,ts,'Upward geothermal heat flux at sea floor',/archive/John.Krasting/fre/FMS2024.02_OM5_20240724/CM4.5v01_om5b06_piC_noBLING/gfdl.ncrc5-intel23-prod-openmp/pp/ocean_monthly/[ocean_monthly.static.nc](http://ocean_monthly.static.nc/)

Option 3:

For the GFDL runs: what-if variable_id is empty or descriptive to denote it's a static dataset.
This means the user is giving an example notebook to search for the static dataset (search does not container variable_id, but it WILL contain realm and frequency). The catalog builder API or an example notebook will show how to look for variables in the returned xarray dataset.

Pros: catalog builder remains light-weight, but there are helper utilities to explore multiple variables in a single dataset. No redundant data arrays in the xarray obj.
Cons: MDTF will NOT have access to standard_name for these statics from catalogs.
Workaround - ensure the use of GFDL-CMOR tables to identify the standard names.

dev,,fre,CM4.5v01_om5b06_piC_noBLING,mon,ocean_monthly,fx,,,static,,,gfdl.ncrc5-intel23-prod-openmp,,ts,'',/archive/John.Krasting/fre/FMS2024.02_OM5_20240724/CM4.5v01_om5b06_piC_noBLING/gfdl.ncrc5-intel23-prod-openmp/pp/ocean_monthly/ocean_monthly.static.nc

[aradhakrishnanGFDL]

Option 3 was agreed upon based on input received at the GFDL analysis subgroup meeting today, in terms of how the catalog looks and how it can be used.
This will keep the catalog builder light-weight, and also give the user the ability to explore the dataset with xarray and pick whichever data variable interests them from the static variable.

variable_id = "fixed"
frequency = "fx"
table_id = "fx"
realm = "atmos" #or land etc

So, the static dataset for different realms will be presented as this key in the returned dictionary from intake-esm, following option 3.

fre.CM4.5v01_om5b06_piC_noBLING.fx.fx.ocean_monthly
fre.CM4.5v01_om5b06_piC_noBLING.fx.fx.land

and so on.

If we still want variable_id = "ocean_monthly.fixed" to be more descriptive, we can do that. This was an idea from @apreucil.
I realize that the variable_id has no place in the dictionary key, so searching by realm (pp directory/component) and the frequency will still give us what we want. If the user searches by variable_id, then searching by realm won't be necessary if we have variable_id = "ocean_monthly.fixed"

[aradhakrishnanGFDL]

closing discussing, option 3 will be implemented and hope this is compatible with the MDTF. If not..please open an issue @wrongkindofdoctor