passing information among different tasks

[uturuncoglu]

I wonder if there is a way of updating information read from the config.yaml file. In my case, workflow has a task that downloads forcing data and I need to point those data files when I am creating cdeps stream namelist file (defined by stream_data_files). This is could be also used to update other informations such as ESMF Mesh file (stream_mesh_file) etc. Maybe it could be another way to pass information between the tasks but I am not sure since current implementation of cdeps.atm_stream() does not accept any arguments to use. Do we need to extend it to accept options? or maybe dynamically updating (or adding entries) to config.yaml would be the option. Not sure since then the result config.yaml could end up a different one since it is dynamically changing along the way and it might not be the safest way and might be hard to debug in case of any issue. Anyway, any suggestion might help.

[maddenp-noaa]

Hi @uturuncoglu, thanks for starting this discussion. I have some questions:

1. Are the forcing-data filenames deterministic based on some aspects of the experiment's configuration? That is, is it possible to calculate in advance what the names will be, and use those names both to a) do the download, and b) modify the CDEPS / Coastal YAML config (I'm guessing that this is what we're talking about) with those values? I can imagine that there might be value in being able to modify the YAML and run the part of the experiment that uses those files while skipping the step that downloads the files -- maybe the files can be copied manually into place in some cases, like for debugging.
2. What language is your workflow code written in? If it's bash, you can use the uw config realize mode and action to programmatically set the filenames in the YAML config once you know what the filenames are. If you're in Python, you can use uwtools.api.config.realize() to do the same.
3. You could potentially opt to have the YAML config contain invariant, canonical names for the files, and rely on either an upstream task or a manual user action to create symbolic links. For example, in a different, imaginary application that consumes an NWP analysis file, the config might contain an item like atm_filename: atm.nc and another task (or a user) might ln -s gfs.t00z.atmanl.nc atm.nc for a specific run. That would delegate the knowledge about the actual filename to the process/person that knows it, relying on the convention that there will always be a symlink with a predictable name.

Do we need to extend it to accept options?

Probably not: The design of the UW drivers is such that they are expected to get any needed information from the config that the driver is instantiated with, which is also schema-checked for correctness.

or maybe dynamically updating (or adding entries) to config.yaml would be the option.

I think this is the right approach. Depending on your workflow, and especially whether it processes any tasks in parallel, you might need to consider making a copy of the main config and editing only the copy, perhaps in the run directory for a specific task.

Those are just some initial thoughts. We'd be happy to look at code, too, and try to help come up with good ideas.

[uturuncoglu]

Thanks for your help. Here are my comments,

1. I could fix the name of the files. I am using external Python module called Herbie and once I download data, I am subsetting them and I could rename them in a certain way (I also tried to combine them but it is slow, so I need to find a more efficient way). So, maybe YAML configuration file might have some fixed file names and variable names. Probably, I could update variable names without any issue.
2. I am trying to implement everything in Python. So, uwtools.api.config.realize() maybe the way to go if I need.
3. Yes, I agree. I think I could follow this approach. Let me try.

Thanks for great ideas. I just need to test them to see how this will work in the real workflow configuration. I'll update you once I test it.

[maddenp-noaa]

Hi @uturuncoglu. I just wanted to let you know that I merged a small PR updating the coastal.py driver for compatibility with the uwtools version 2.5.0 that we just released. The changes are trivial, but I thought I should let you know.

[uturuncoglu]

@maddenp-noaa Thanks. Let me try with that version too. I'll update you about it, if I have any issue.
BTW, is there any news about generating driver specific ufs.configure and model_configure files.

[uturuncoglu]

@maddenp-noaa I circle back to my initial question again. The workflow that I am working on gets input forcing and creates CDEPS namelist files based on it. The issue is that there are some options in the CDEPS namelist files that are input file specific and it is hard to know those options in advance since we are subsetting the forcing file based on the domain file. For example, nx_global and ny_global are the lat and lon dimensions of the input file that could change in every case. So, I wonder if we could bring some capability to overwrite options given by config.yaml file. This could be also used to define some defaults since some options are same for all the cases.

[maddenp-noaa]

@uturuncoglu I feel confident there's a clean solution, but it would probably be easier to look at your code together. Maybe we could schedule some time soon to do a screen-sharing sessions to look at what you've got. If you like, please feel free to invite me -- you should be able to see my calendar with your NOAA email and e.g. invite me to a Meet.

[uturuncoglu]

@maddenp-noaa Sure. Thanks for your help. Let me find a time.

[uturuncoglu]

@maddenp-noaa I have just send an invitation. Please let me know if it not working for you and we could change it.

[maddenp-noaa]

@uturuncoglu I had a few thoughts after our Meet discussion:

1. I think that we would not currently have access to the path to the original config file to get its timestamp to implement a make-like behavior where output would be regenerated based on a change to the config file. I think I can easily add this to uwtools, though, so that Config objects (like the YAMLConfig that drivers receive) would have an attribute, perhaps .config_file, that would provide the needed path to the file. I'll see about implementing this so that we have it available when we need it later. I think this could be helpful for applications other than Coastal, too.

2. Although the task methods in the drivers that are meant to be callable from the CLI should accept no arguments and only be controlled via information in the driver's config, it's definitely ok to define helper tasks that do take arguments. The names of these method should start with a _ to prevent them from showing up in the help output. For example, I can add a task method like

@task
def _helper(self, n: int):
    """
    A helper task.
    """
    path = Path("/path/to/file-%s" % n)
    yield "Helper %s" % n
    yield asset(path, path.is_file)
    yield None
    path.touch()

to the CDEPS class, but it will not be advertised via the CLI:

$ uw cdeps -h
usage: uw cdeps [-h] [--version] [--show-schema] TASK ...
...
Positional arguments:
  TASK
    atm
      The data atmosphere configuration with all required content
    atm_nml
      The data atmosphere Fortran namelist file (datm_in)
    atm_stream
      The data atmosphere stream config file (datm.streams)
    ocn
      The data ocean configuration with all required content
    ocn_nml
      The data ocean Fortran namelist file (docn_in)
    ocn_stream
      The data ocean stream config file (docn.streams)
    validate
      Validate the UW driver config

But another task could e.g. yield [_helper(n) for n in range(8)] to declare a bunch of _helper tasks as dependencies. Feel free to create internal-use tasks like these if it helps break down the workflow in a cleaner way. We just want those tasks that can be called directly from the CLI to take no arguments and only rely on the driver's config.

3. I put together a little demo of some of the ideas we talked about, attached as demo.py.txt (rename to demo.py before using). It extracts specific 2D fields from HRRR GRIB files in AWS S3, and reports the dimensions of the grids. You can run it like

$ conda create -y -n demo -c ufs-community cfgrib requests uwtools xarray
$ conda activate demo
$ rm -rf /tmp/demo
$ $ iotaa demo.py report
[2024-12-17T22:57:42] INFO    GRIB files 20241201 0 [3, 6]: Initial state: Not Ready
[2024-12-17T22:57:42] INFO    GRIB files 20241201 0 [3, 6]: Checking requirements
[2024-12-17T22:57:42] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf03.grib2: Initial state: Not Ready
[2024-12-17T22:57:42] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf03.grib2: Checking requirements
[2024-12-17T22:57:42] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf03.grib2.idx: Initial state: Not Ready
[2024-12-17T22:57:42] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf03.grib2.idx: Checking requirements
[2024-12-17T22:57:42] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf03.grib2.idx: Requirement(s) ready
[2024-12-17T22:57:42] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf03.grib2.idx: Executing
[2024-12-17T22:57:42] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf03.grib2.idx: Fetching https://noaa-hrrr-bdp-pds.s3.amazonaws.com/hrrr.20241201/conus/hrrr.t00z.wrfprsf03.grib2.idx
[2024-12-17T22:57:43] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf03.grib2.idx: Wrote /tmp/demo/hrrr.t00z.wrfprsf03.grib2.idx
[2024-12-17T22:57:43] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf03.grib2.idx: Final state: Ready
[2024-12-17T22:57:43] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf03.grib2: Requirement(s) ready
[2024-12-17T22:57:43] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf03.grib2: Executing
[2024-12-17T22:57:43] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf03.grib2: Fetching https://noaa-hrrr-bdp-pds.s3.amazonaws.com/hrrr.20241201/conus/hrrr.t00z.wrfprsf03.grib2
[2024-12-17T22:57:43] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf03.grib2: Wrote /tmp/demo/hrrr.t00z.wrfprsf03.grib2
[2024-12-17T22:57:43] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf03.grib2: Final state: Ready
[2024-12-17T22:57:44] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf06.grib2: Initial state: Not Ready
[2024-12-17T22:57:44] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf06.grib2: Checking requirements
[2024-12-17T22:57:44] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf06.grib2.idx: Initial state: Not Ready
[2024-12-17T22:57:44] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf06.grib2.idx: Checking requirements
[2024-12-17T22:57:44] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf06.grib2.idx: Requirement(s) ready
[2024-12-17T22:57:44] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf06.grib2.idx: Executing
[2024-12-17T22:57:44] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf06.grib2.idx: Fetching https://noaa-hrrr-bdp-pds.s3.amazonaws.com/hrrr.20241201/conus/hrrr.t00z.wrfprsf06.grib2.idx
[2024-12-17T22:57:44] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf06.grib2.idx: Wrote /tmp/demo/hrrr.t00z.wrfprsf06.grib2.idx
[2024-12-17T22:57:44] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf06.grib2.idx: Final state: Ready
[2024-12-17T22:57:44] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf06.grib2: Requirement(s) ready
[2024-12-17T22:57:44] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf06.grib2: Executing
[2024-12-17T22:57:44] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf06.grib2: Fetching https://noaa-hrrr-bdp-pds.s3.amazonaws.com/hrrr.20241201/conus/hrrr.t00z.wrfprsf06.grib2
[2024-12-17T22:57:45] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf06.grib2: Wrote /tmp/demo/hrrr.t00z.wrfprsf06.grib2
[2024-12-17T22:57:45] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf06.grib2: Final state: Ready
[2024-12-17T22:57:45] INFO    GRIB files 20241201 0 [3, 6]: Requirement(s) ready
[2024-12-17T22:57:45] INFO    GRIB files 20241201 0 [3, 6]: Executing
Field 'TMP' at level '2 m above ground' in 20241201 0 3 has dimensions (1799, 1059)
Field 'TMP' at level '2 m above ground' in 20241201 0 6 has dimensions (1799, 1059)

I think it demonstrates some of the ideas we talked about, and shows how public tasks with no arguments, private helper tasks with arguments, and helper functions can work together to model a more complicated workflow, with each lower-level task communicating information that it is responsible for upwards, and how higher-level tasks can use that information. I hope it's useful.

demo.py.txt

[maddenp-noaa]

1. Oh, I'm on the wrong branch. Let me try again.
2. ^^^
3. Let me see how far I can get without the data files. I can probably just create empty files to get the run directory created, without running the components.

[maddenp-noaa]

I made progress toward at least getting a provisioned run directory, even if I won't be able to do an actual run. I created empty files for those in the coastal.links: block so the driver has something to link to, but the file hgrid.gr3 needs to contain real data, as read_hgrid() wants to read it at configuration time. I could probably mock something up, but if that's a small file and doesn't contain any sensitive info, maybe you could attach it (and any others you know that the driver needs at config time, rather than at run time) here, I could use those. I think that I might be able to follow along with your development, and possibly consult on design questions, if I can at least produce a provisioned run directory.

[uturuncoglu]

@maddenp-noaa Sorry for late response. Thanks for your help. I am attaching both grid file (hgird and vgrid) to here.
grids.tar.gz

[maddenp-noaa]

@uturuncoglu I had some time yesterday to catch up on this and was able to, I think, get the provisioned_rundir task and its dependencies to complete successfully. I'm attaching the output of uw execute --module coastal.py --classname Coastal --task provisioned_rundir --config-file coastal.yaml --cycle 2024-08-05T12 --batch here: log.txt

You can see the code here. I made significant changes to coastal.py, more minor changes to coastal.yaml and schism.jsonschema, and fairly trivial (but useful, I think) changes to some of the .py modules under utils/. More about these below.

A main issue, which I solved (at least for this specific configuration) in the forcing_data() and mesh_files() task functions (formerly _data_retrieve() and _create_mesh()), is that the tasks were performing their work before declaring what it would be. The same thing is still a problem in schism_*_inputs() functions. This amounts to a circular definition: It says that the correct/expected output of the task is whatever the task creates. I fixed this be declaring that forcing_data() will create the combined.nc file, and mesh_file() will create two files: mesh.nc which is the data, and cdeps.yaml, which is then used to create the CDEPS driver object and contains information from the call to gen_grid_def().

In the case of these two tasks, it was clear what the output would be. Hopefully the same can be done for the schism_*_inputs() functions. One thing worth considering: Rather than having utility functions define their filenames, high-level code can tell them what file to create. This shouldn't make any different to the utility functions, but then the higher-level code will know what to expect from the function, and the filenames will be defined in only one place -- unlike now where e.g. the driver code needs to know that gen_grid_def() will create a file called mesh.nc, but gen_grid_def() has to duplicate the string "mesh.nc", and both pieces of code have to be kept in sync. By factoring this fact out, either into the driver code if it is something unlikely that a user will ever want to configure, or even to the YAML config if it is configurable, there's a single source of truth for this, and no doubt about what the utility function will create. (Sometimes a utility function or external program needs to create files with time or other information embedded in the filename. In that case, since the function/program is being configured by the driver, the driver should also be able to construct the filename from its config data and knowledge about the function; or the logic used to construct the filename could be factored out to a separate utility function that could be called both by the driver and by the gets-work-done utility function.)

Some other changes:

• Since it's mesh_file() that needs the combined.nc data, mesh_file() now depends on forcing_data() instead of having the latter declared a dependency of provisioned_rundir(). This enforces the order of execution correctly, and it lets mesh_file() get the name of the netCDF file from the asset yielded by forcing_data(), here. Since forcing_data() yields a single asset, calling refs() on its return value gets the "ref" part of the asset(ref, ready) yielded here. This way, mesh_file() does not have to know the name combined.nc -- it can get that information from mesh_file() so that, if that name changed, only mesh_file() would have to be updated. mesh_file() yields two assets, so they're yielded in a dict here so that the customized CDEPS driver config can be referenced here.
• I arranged for Herbie and the external program ESMF_Scrip2Unstruct to write their console output to log files combined.log and mesh.log in the run directory, instead of to stdout. This keeps the log output from the driver clean, with every line properly timestamped. (It would also be possible to capture the output and correctly log it inline with the other log output -- I can help with that if you're interested. But separate log files have their value, too.)
• In addition to running coastal.yaml through a tidy program, I updated it so that the start_day etc. for the SCHISM namelist file are calculated directly in the YAML, here. For cycle-based drivers, uwtools arranges for a cycle object to be available to Jinja2 when the driver's config is loaded, so we don't need extra Python code for this.
• I ran a tidy program on schism.jsonschema, and added some addition requirements that seemed to me like they would be necessary, like start_day, dt, etc. Since these are {{ expressions }} in the template, I don't see how the config could be correct without these values -- the template could not be rendered.
• I changed some print() statements to logging.*() functions, and used logging.debug() in a few places instead of logging.info() where the log info seemed like maybe too much for a normal run. Please feel free to change back to .info() if you do want to always see those; but you can also run uw with -v / --verbose to see all the possible log messages.

I hope the code and techniques make sense. We could go over anything you have doubts about in a Google Meet if that would be helpful.

I wanted to do PR for you but, unfortunately, I neglected to update my code with your latest changes before I started working on it yesterday, so the merge I'd need to do for a PR is a little messy. I'm happy to do it, though, if you like these suggestions and would like to merge them via a PR. Otherwise, the code is there in my fork for you to copy/borrow from. You can see the full diff between my branch and where I started yesterday here.

[uturuncoglu]

@maddenp-noaa Thanks you very much. Let me check those improvements and try to port into my branch. I'll update you about it. Thanks again.

[uturuncoglu]

@maddenp-noaa JFYI, I pushed the changes to my development branch. I still need to fix the schsim functions. Thanks again for your help. If you want we could close this discussion and I could mark the last conversation as answer.

[maddenp-noaa]

Thanks, yes, please go ahead and close this conversation -- and feel free to open a new one when there's a new topic!