passing information among different tasks

[uturuncoglu]

I wonder if there is a way of updating information read from the config.yaml file. In my case, workflow has a task that downloads forcing data and I need to point those data files when I am creating cdeps stream namelist file (defined by stream_data_files). This is could be also used to update other informations such as ESMF Mesh file (stream_mesh_file) etc. Maybe it could be another way to pass information between the tasks but I am not sure since current implementation of cdeps.atm_stream() does not accept any arguments to use. Do we need to extend it to accept options? or maybe dynamically updating (or adding entries) to config.yaml would be the option. Not sure since then the result config.yaml could end up a different one since it is dynamically changing along the way and it might not be the safest way and might be hard to debug in case of any issue. Anyway, any suggestion might help.

[maddenp-noaa]

Hi @uturuncoglu, thanks for starting this discussion. I have some questions:

1. Are the forcing-data filenames deterministic based on some aspects of the experiment's configuration? That is, is it possible to calculate in advance what the names will be, and use those names both to a) do the download, and b) modify the CDEPS / Coastal YAML config (I'm guessing that this is what we're talking about) with those values? I can imagine that there might be value in being able to modify the YAML and run the part of the experiment that uses those files while skipping the step that downloads the files -- maybe the files can be copied manually into place in some cases, like for debugging.
2. What language is your workflow code written in? If it's bash, you can use the uw config realize mode and action to programmatically set the filenames in the YAML config once you know what the filenames are. If you're in Python, you can use uwtools.api.config.realize() to do the same.
3. You could potentially opt to have the YAML config contain invariant, canonical names for the files, and rely on either an upstream task or a manual user action to create symbolic links. For example, in a different, imaginary application that consumes an NWP analysis file, the config might contain an item like atm_filename: atm.nc and another task (or a user) might ln -s gfs.t00z.atmanl.nc atm.nc for a specific run. That would delegate the knowledge about the actual filename to the process/person that knows it, relying on the convention that there will always be a symlink with a predictable name.

Do we need to extend it to accept options?

Probably not: The design of the UW drivers is such that they are expected to get any needed information from the config that the driver is instantiated with, which is also schema-checked for correctness.

or maybe dynamically updating (or adding entries) to config.yaml would be the option.

I think this is the right approach. Depending on your workflow, and especially whether it processes any tasks in parallel, you might need to consider making a copy of the main config and editing only the copy, perhaps in the run directory for a specific task.

Those are just some initial thoughts. We'd be happy to look at code, too, and try to help come up with good ideas.

[uturuncoglu]

Thanks for your help. Here are my comments,

1. I could fix the name of the files. I am using external Python module called Herbie and once I download data, I am subsetting them and I could rename them in a certain way (I also tried to combine them but it is slow, so I need to find a more efficient way). So, maybe YAML configuration file might have some fixed file names and variable names. Probably, I could update variable names without any issue.
2. I am trying to implement everything in Python. So, uwtools.api.config.realize() maybe the way to go if I need.
3. Yes, I agree. I think I could follow this approach. Let me try.

Thanks for great ideas. I just need to test them to see how this will work in the real workflow configuration. I'll update you once I test it.

[maddenp-noaa]

Hi @uturuncoglu. I just wanted to let you know that I merged a small PR updating the coastal.py driver for compatibility with the uwtools version 2.5.0 that we just released. The changes are trivial, but I thought I should let you know.

[uturuncoglu]

@maddenp-noaa Thanks. Let me try with that version too. I'll update you about it, if I have any issue.
BTW, is there any news about generating driver specific ufs.configure and model_configure files.

[uturuncoglu]

@maddenp-noaa I circle back to my initial question again. The workflow that I am working on gets input forcing and creates CDEPS namelist files based on it. The issue is that there are some options in the CDEPS namelist files that are input file specific and it is hard to know those options in advance since we are subsetting the forcing file based on the domain file. For example, nx_global and ny_global are the lat and lon dimensions of the input file that could change in every case. So, I wonder if we could bring some capability to overwrite options given by config.yaml file. This could be also used to define some defaults since some options are same for all the cases.

[maddenp-noaa]

@uturuncoglu I feel confident there's a clean solution, but it would probably be easier to look at your code together. Maybe we could schedule some time soon to do a screen-sharing sessions to look at what you've got. If you like, please feel free to invite me -- you should be able to see my calendar with your NOAA email and e.g. invite me to a Meet.

[uturuncoglu]

@maddenp-noaa Sure. Thanks for your help. Let me find a time.

[uturuncoglu]

@maddenp-noaa I have just send an invitation. Please let me know if it not working for you and we could change it.

[maddenp-noaa]

@uturuncoglu I had a few thoughts after our Meet discussion:

1. I think that we would not currently have access to the path to the original config file to get its timestamp to implement a make-like behavior where output would be regenerated based on a change to the config file. I think I can easily add this to uwtools, though, so that Config objects (like the YAMLConfig that drivers receive) would have an attribute, perhaps .config_file, that would provide the needed path to the file. I'll see about implementing this so that we have it available when we need it later. I think this could be helpful for applications other than Coastal, too.

2. Although the task methods in the drivers that are meant to be callable from the CLI should accept no arguments and only be controlled via information in the driver's config, it's definitely ok to define helper tasks that do take arguments. The names of these method should start with a _ to prevent them from showing up in the help output. For example, I can add a task method like

@task
def _helper(self, n: int):
    """
    A helper task.
    """
    path = Path("/path/to/file-%s" % n)
    yield "Helper %s" % n
    yield asset(path, path.is_file)
    yield None
    path.touch()

to the CDEPS class, but it will not be advertised via the CLI:

$ uw cdeps -h
usage: uw cdeps [-h] [--version] [--show-schema] TASK ...
...
Positional arguments:
  TASK
    atm
      The data atmosphere configuration with all required content
    atm_nml
      The data atmosphere Fortran namelist file (datm_in)
    atm_stream
      The data atmosphere stream config file (datm.streams)
    ocn
      The data ocean configuration with all required content
    ocn_nml
      The data ocean Fortran namelist file (docn_in)
    ocn_stream
      The data ocean stream config file (docn.streams)
    validate
      Validate the UW driver config

But another task could e.g. yield [_helper(n) for n in range(8)] to declare a bunch of _helper tasks as dependencies. Feel free to create internal-use tasks like these if it helps break down the workflow in a cleaner way. We just want those tasks that can be called directly from the CLI to take no arguments and only rely on the driver's config.

3. I put together a little demo of some of the ideas we talked about, attached as demo.py.txt (rename to demo.py before using). It extracts specific 2D fields from HRRR GRIB files in AWS S3, and reports the dimensions of the grids. You can run it like

$ conda create -y -n demo -c ufs-community cfgrib requests uwtools xarray
$ conda activate demo
$ rm -rf /tmp/demo
$ $ iotaa demo.py report
[2024-12-17T22:57:42] INFO    GRIB files 20241201 0 [3, 6]: Initial state: Not Ready
[2024-12-17T22:57:42] INFO    GRIB files 20241201 0 [3, 6]: Checking requirements
[2024-12-17T22:57:42] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf03.grib2: Initial state: Not Ready
[2024-12-17T22:57:42] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf03.grib2: Checking requirements
[2024-12-17T22:57:42] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf03.grib2.idx: Initial state: Not Ready
[2024-12-17T22:57:42] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf03.grib2.idx: Checking requirements
[2024-12-17T22:57:42] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf03.grib2.idx: Requirement(s) ready
[2024-12-17T22:57:42] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf03.grib2.idx: Executing
[2024-12-17T22:57:42] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf03.grib2.idx: Fetching https://noaa-hrrr-bdp-pds.s3.amazonaws.com/hrrr.20241201/conus/hrrr.t00z.wrfprsf03.grib2.idx
[2024-12-17T22:57:43] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf03.grib2.idx: Wrote /tmp/demo/hrrr.t00z.wrfprsf03.grib2.idx
[2024-12-17T22:57:43] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf03.grib2.idx: Final state: Ready
[2024-12-17T22:57:43] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf03.grib2: Requirement(s) ready
[2024-12-17T22:57:43] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf03.grib2: Executing
[2024-12-17T22:57:43] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf03.grib2: Fetching https://noaa-hrrr-bdp-pds.s3.amazonaws.com/hrrr.20241201/conus/hrrr.t00z.wrfprsf03.grib2
[2024-12-17T22:57:43] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf03.grib2: Wrote /tmp/demo/hrrr.t00z.wrfprsf03.grib2
[2024-12-17T22:57:43] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf03.grib2: Final state: Ready
[2024-12-17T22:57:44] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf06.grib2: Initial state: Not Ready
[2024-12-17T22:57:44] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf06.grib2: Checking requirements
[2024-12-17T22:57:44] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf06.grib2.idx: Initial state: Not Ready
[2024-12-17T22:57:44] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf06.grib2.idx: Checking requirements
[2024-12-17T22:57:44] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf06.grib2.idx: Requirement(s) ready
[2024-12-17T22:57:44] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf06.grib2.idx: Executing
[2024-12-17T22:57:44] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf06.grib2.idx: Fetching https://noaa-hrrr-bdp-pds.s3.amazonaws.com/hrrr.20241201/conus/hrrr.t00z.wrfprsf06.grib2.idx
[2024-12-17T22:57:44] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf06.grib2.idx: Wrote /tmp/demo/hrrr.t00z.wrfprsf06.grib2.idx
[2024-12-17T22:57:44] INFO    Index file /tmp/demo/hrrr.t00z.wrfprsf06.grib2.idx: Final state: Ready
[2024-12-17T22:57:44] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf06.grib2: Requirement(s) ready
[2024-12-17T22:57:44] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf06.grib2: Executing
[2024-12-17T22:57:44] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf06.grib2: Fetching https://noaa-hrrr-bdp-pds.s3.amazonaws.com/hrrr.20241201/conus/hrrr.t00z.wrfprsf06.grib2
[2024-12-17T22:57:45] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf06.grib2: Wrote /tmp/demo/hrrr.t00z.wrfprsf06.grib2
[2024-12-17T22:57:45] INFO    GRIB file /tmp/demo/hrrr.t00z.wrfprsf06.grib2: Final state: Ready
[2024-12-17T22:57:45] INFO    GRIB files 20241201 0 [3, 6]: Requirement(s) ready
[2024-12-17T22:57:45] INFO    GRIB files 20241201 0 [3, 6]: Executing
Field 'TMP' at level '2 m above ground' in 20241201 0 3 has dimensions (1799, 1059)
Field 'TMP' at level '2 m above ground' in 20241201 0 6 has dimensions (1799, 1059)

I think it demonstrates some of the ideas we talked about, and shows how public tasks with no arguments, private helper tasks with arguments, and helper functions can work together to model a more complicated workflow, with each lower-level task communicating information that it is responsible for upwards, and how higher-level tasks can use that information. I hope it's useful.

demo.py.txt

[maddenp-noaa]

@uturuncoglu I was able to do the CSTLS build and create the conda environment on Hera. Next:

1. In your recipe, cd ufs-coastal-app/ush doesn't work for me as ush/ does not seem to exist. Do I just need to create that?
2. Should I get coastal.yaml, coastal.py, etc. from the main uw-coupled-drivers repo, or do you have an updated fork that I should clone to get your latest versions of those?
3. I guess I'll need a copy of the staged SCHISM data that the driver requires, too. Do you have an account on Hera where you could copy it? Or other ideas about how we can transfer data? A shared Google Drive link maybe (you could email that to me separately).

[uturuncoglu]

1 . It must be in there - https://github.com/oceanmodeling/ufs-coastal-app/tree/feature/workflow Could you double check your branch? It must be feature/workflow.
2. All the required files are under ush which also includes coastal.yaml and coastal.py. The ones in uw-coupled-drivers will not work since I add more staff to it.
3. I have no account on Hera. If you want I could put them into cloud but it would take some time to do it.

[maddenp-noaa]

1. Oh, I'm on the wrong branch. Let me try again.
2. ^^^
3. Let me see how far I can get without the data files. I can probably just create empty files to get the run directory created, without running the components.

[maddenp-noaa]

I made progress toward at least getting a provisioned run directory, even if I won't be able to do an actual run. I created empty files for those in the coastal.links: block so the driver has something to link to, but the file hgrid.gr3 needs to contain real data, as read_hgrid() wants to read it at configuration time. I could probably mock something up, but if that's a small file and doesn't contain any sensitive info, maybe you could attach it (and any others you know that the driver needs at config time, rather than at run time) here, I could use those. I think that I might be able to follow along with your development, and possibly consult on design questions, if I can at least produce a provisioned run directory.

[uturuncoglu]

@maddenp-noaa Sorry for late response. Thanks for your help. I am attaching both grid file (hgird and vgrid) to here.
grids.tar.gz