Community Engagement Improves UW Tools

[maddenp-noaa]

It's no accident that the Unified Workflow Tools repo lives in the ufs-community GitHub organization, or that we publish our conda packages to a ufs-community channel: We are building tools to address community needs, and community engagement -- via discussions, feature requests, and, yes, even bug reports -- is crucial to growing the utility of uwtools in solving recurring NWP workflow problems.

We've already received some excellent suggestions from users that have led to our implementing valuable new uwtools features. Here are a few.

Accepting uw config realize update values on stdin

Like most tools in the Unix tradition, uw accepts input piped in via stdin as an alternative to disk files. Early on, uw config realize supported providing its input config via stdin, but required that update values come from a file. So the following has long worked: Given

update.yaml

time:
  yyyymmdd: 20240905

we can update the Fortran namelist config provided on stdin with the values from the YAML file:

$ echo "&time yyyymmdd=-1 /" | uw config realize --input-format nml --output-format nml --update-file update.yaml 
&time
    yyyymmdd = 20240905
/

However, an important (and probably even more common) use case involves an input config file, updated with values read from stdin. Based on user feedback, we added support for this to uwtools, activated on the CLI by use of the --update-format flag. Given

config.nml

&time
    yyyymmdd = -1
/

we can update the Fortran namelist in config.nml with values supplied on stdin:

$ echo "time: {yyyymmdd: 20240905}" | uw config realize --input-file config.nml --output-format nml --update-format yaml
&time
    yyyymmdd = 20240905
/

The uw config realize --total flag

The default behavior of uw config realize is to render as many embedded Jinja2 expressions as possible, leaving others intact, potentially to be rendered by later processing. For example, a config might define the name of an atmospheric analysis file, based on a prefix read from an environment variable:

config.yaml

atmanl: "{{ pre }}.t{{ hh }}z.atmanl.nc"
pre: "{{ 'PRODUCT' | env }}"

If we already know that GFS analysis will be used, we might want to partially realize the config with that information:

$ PRODUCT=gfs uw config realize --input-file config.yaml --output-file partial.yaml

partial.yaml

atmanl: '{{ pre }}.t{{ hh }}z.atmanl.nc'
pre: gfs

Later, partial.yaml could be fully realized by providing cycle information:

$ echo "hh: 12" | uw config realize --input-file partial.yaml --update-format yaml --output-format yaml
atmanl: gfs.t12z.atmanl.nc
pre: gfs
hh: 12

The above demonstrates why partial rendering can be useful. However, we often want to fully realize a config immediately:

$ echo "hh: 12" | PRODUCT=gfs uw config realize --input-file config.yaml --update-format yaml --output-format yaml
atmanl: gfs.t12z.atmanl.nc
pre: gfs
hh: 12

But what if we've introduced an error that prevents this? A uwtools user suggested to us that a way to detect an incompletely realized config and fail early could save time and make troubleshooting easier. We added the --total flag to provide this mechanism.

For example:

$ echo "h: 12" | PRODUCT=gfs uw config realize --input-file config.yaml --update-format yaml --output-format yaml --total
[2024-09-05T18:00:56]    ERROR Config could not be realized. Try with --values-needed for details.

$ echo $?
1

$ echo "h: 12" | PRODUCT=gfs uw config realize --input-file config.yaml --update-format yaml --output-format yaml --values-needed
[2024-09-05T18:01:00]     INFO Keys that are complete:
[2024-09-05T18:01:00]     INFO   pre
[2024-09-05T18:01:00]     INFO   h
[2024-09-05T18:01:00]     INFO 
[2024-09-05T18:01:00]     INFO Keys with unrendered Jinja2 variables/expressions:
[2024-09-05T18:01:00]     INFO   atmanl: {{ pre }}.t{{ hh }}z.atmanl.nc

Here, --total causes uw to exit with error status (which could be used to terminate e.g. a Bash script calling uw) after advising us to use the --values-needed flag to report unrendered Jinja2 expressions. The --values-needed output makes the typo (h instead of hh) easy to spot.

Support for Jinja2 macros in uw template render

Jinja2 provides support for macros, which can be accessed in a template via its import directive. Guided by a bug report filed by a user, we added support for importing macros in templates processed by uw template render.

For example, given a file containing a Jinja2 macro:

macros.jinja2

{% macro gfs_fcst_url(yyyymmdd, hh, leadtime) -%}
https://ftpprd.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.{{ yyyymmdd }}/{{ hh }}/atmos/gfs.t{{ hh }}z.pgrb2b.0p25.f{{ "%03d" % leadtime }}
{%- endmacro %}

and a template file referencing it:

template.jinja2

#!/bin/bash

{% import "macros.jinja2" as m -%}
{% for n in range(7) -%}
wget {{ m.gfs_fcst_url(yyyymmdd, hh, n) }}
{% endfor %}

we can render the template into a script for downloading some GFS forecast files:

$ uw template render --input-file template.jinja2 yyyymmdd=20240905 hh=12
#!/bin/bash

wget https://ftpprd.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.20240905/12/atmos/gfs.t12z.pgrb2b.0p25.f000
wget https://ftpprd.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.20240905/12/atmos/gfs.t12z.pgrb2b.0p25.f001
wget https://ftpprd.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.20240905/12/atmos/gfs.t12z.pgrb2b.0p25.f002
wget https://ftpprd.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.20240905/12/atmos/gfs.t12z.pgrb2b.0p25.f003
wget https://ftpprd.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.20240905/12/atmos/gfs.t12z.pgrb2b.0p25.f004
wget https://ftpprd.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.20240905/12/atmos/gfs.t12z.pgrb2b.0p25.f005
wget https://ftpprd.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.20240905/12/atmos/gfs.t12z.pgrb2b.0p25.f006

The macros.jinja2 file could be reused by importing it into other templates, keeping knowledge about the upstream location of these files DRY.

(Note that template.jinja2 could find macros.jinja2 because they were located in the same directory. The optional --search-path option can be used to provide a colon-delimited list of directories to search for Jinja2 macro files.)

File copy/link tool

A common need in NWP workflows involves copying or linking standard sets of files into run directories. Based on a feature request from a user, we added this functionality to uwtools.

While a one-off implementation isn't difficult to achieve in, say, Bash, uwtools support means that

• File source and destination paths can be pre-processed with tools like uw config realize or uw template render.
• YAML snippets defining common file sets can be shared between configs or even applications.
• Jinja2 techniques can be used to keep configs DRY.
• First-class support for cycle and/or leadtime values is trivially available.

For example, we might need to use cycle and leadtime information in various ways like in:

config.yaml

base: /scratch2/BMC/public/data/grids/gfs

fff: "{{ '%03d' % (leadtime.total_seconds() / 3600) }}"
ffffff: "{{ '%06d' % (leadtime.total_seconds() / 3600) }}"
hh: "{{ cycle.strftime('%H') }}"
ord: "{{ cycle.strftime('%j') }}"
pre: "gfs.{{ cycle.strftime('%Y%m%d') }}.{{ hh }}"
yy: "{{ cycle.strftime('%y') }}"

anl:
  anl/{{ pre }}.nc: "{{ base }}/anl/netcdf/{{ yy }}{{ ord }}{{ hh }}00.gfs.t{{ hh }}z.sfcanl.nc"
fcst:
  fcst/{{ pre }}.f{{ fff }}.grib2: "{{ base }}/0p25deg/grib2/{{ yy }}{{ ord }}{{ hh }}{{ ffffff }}"

We can symlink an analysis file by using the link action and selecting the anl block:

$ uw file link --config-file config.yaml --cycle 2024-09-05T12 --target-dir /tmp/$USER --leadtime 6 anl
[2024-09-05T20:48:20]     INFO Validating config against internal schema: files-to-stage
[2024-09-05T20:48:20]     INFO 0 UW schema-validation errors found
[2024-09-05T20:48:20]     INFO File links: Initial state: Not Ready
[2024-09-05T20:48:20]     INFO File links: Checking requirements
[2024-09-05T20:48:20]     INFO Link /tmp/Paul.Madden/anl/gfs.20240905.12.nc -> /scratch2/BMC/public/data/grids/gfs/anl/netcdf/242491200.gfs.t12z.sfcanl.nc: Initial state: Not Ready
[2024-09-05T20:48:20]     INFO Link /tmp/Paul.Madden/anl/gfs.20240905.12.nc -> /scratch2/BMC/public/data/grids/gfs/anl/netcdf/242491200.gfs.t12z.sfcanl.nc: Checking requirements
[2024-09-05T20:48:20]     INFO Link /tmp/Paul.Madden/anl/gfs.20240905.12.nc -> /scratch2/BMC/public/data/grids/gfs/anl/netcdf/242491200.gfs.t12z.sfcanl.nc: Requirement(s) ready
[2024-09-05T20:48:20]     INFO Link /tmp/Paul.Madden/anl/gfs.20240905.12.nc -> /scratch2/BMC/public/data/grids/gfs/anl/netcdf/242491200.gfs.t12z.sfcanl.nc: Executing
[2024-09-05T20:48:20]     INFO Link /tmp/Paul.Madden/anl/gfs.20240905.12.nc -> /scratch2/BMC/public/data/grids/gfs/anl/netcdf/242491200.gfs.t12z.sfcanl.nc: Final state: Ready
[2024-09-05T20:48:20]     INFO File links: Final state: Ready

And we can copy a forecast file by using the copy action and selecting the fcst block:

$ uw file copy --config-file config.yaml --cycle 2024-09-05T12 --target-dir /tmp/$USER --leadtime 6 fcst
[2024-09-05T20:48:47]     INFO Validating config against internal schema: files-to-stage
[2024-09-05T20:48:47]     INFO 0 UW schema-validation errors found
[2024-09-05T20:48:47]     INFO File copies: Initial state: Not Ready
[2024-09-05T20:48:47]     INFO File copies: Checking requirements
[2024-09-05T20:48:47]     INFO Copy /scratch2/BMC/public/data/grids/gfs/0p25deg/grib2/2424912000006 -> /tmp/Paul.Madden/fcst/gfs.20240905.12.f006.grib2: Initial state: Not Ready
[2024-09-05T20:48:47]     INFO Copy /scratch2/BMC/public/data/grids/gfs/0p25deg/grib2/2424912000006 -> /tmp/Paul.Madden/fcst/gfs.20240905.12.f006.grib2: Checking requirements
[2024-09-05T20:48:47]     INFO Copy /scratch2/BMC/public/data/grids/gfs/0p25deg/grib2/2424912000006 -> /tmp/Paul.Madden/fcst/gfs.20240905.12.f006.grib2: Requirement(s) ready
[2024-09-05T20:48:47]     INFO Copy /scratch2/BMC/public/data/grids/gfs/0p25deg/grib2/2424912000006 -> /tmp/Paul.Madden/fcst/gfs.20240905.12.f006.grib2: Executing
[2024-09-05T20:48:47]     INFO Copy /scratch2/BMC/public/data/grids/gfs/0p25deg/grib2/2424912000006 -> /tmp/Paul.Madden/fcst/gfs.20240905.12.f006.grib2: Final state: Ready
[2024-09-05T20:48:47]     INFO File copies: Final state: Ready

The result:

/tmp/Paul.Madden
├── anl
│   └── gfs.20240905.12.nc -> /scratch2/BMC/public/data/grids/gfs/anl/netcdf/242491200.gfs.t12z.sfcanl.nc
└── fcst
    └── gfs.20240905.12.f006.grib2

In a near-future release, we will be renaming the file mode fs ("filesystem") and adding a makedirs action that can be used to create directory structures.

🦾

We'd like to thank our users who have already given us valuable feedback leading to uwtools improvements that we hope will benefit our entire community. If you are reading this, you, too, are part of our community, and we'd love to hear from you in our Discussions or, yes, even in our Issues. Not every potential feature will be a good fit for uwtools, but we have been able to implement those shown here, and several others, so far. Help us make uwtools even better!