Use site-specific lists of recipes in the RTW

doc/sphinx/source/utils/RTW/add_a_recipe.rst

@@ -26,48 +26,78 @@ How to add a recipe to the |RTW|
      [...]
      YYYY-MM-DD HH:MM:SS:sss UTC [12345] INFO    Run was successful
 
-#. Add the recipe to the ``[task parameters]`` section in the
-   ``esmvaltool/utils/recipe_test_workflow/flow.cylc`` file.
+.. _run_a_recipe_in_the_rtw:
 
-   .. hint::
-      If the recipe takes less than 10 minutes to run then it should be added
-      to the ``fast`` option. Recipes that take longer than ten minutes should
-      be added to the ``medium`` option.
+Run a Recipe in the |RTW|
+-------------------------
 
-   .. hint::
-      The line added should follow the format of ``recipe_new_recipe, \``,
-      unless the line is the last one in the list, in which case the line added
-      should follow the format of ``recipe_new_recipe``.
-
-#. If the duration of the recipe is larger than the value specified by the
-   ``execution time limit`` option in the ``[[COMPUTE]]`` section in the
-   aforementioned site-specific ``.cylc`` file, and / or the memory usage of
-   the recipe is larger than the value specified by the ``--mem`` option in the
-   ``[[[directives]]]`` section in the ``[[COMPUTE]]`` section, add a section
-   (in alphabetical order) to this file as shown below (round the duration to
-   the nearest second)::
-
-     [[process<fast=recipe_albedolandcover>]]
-     # Actual: 0m31s, 2.5 GB on 2024-04-08.
-     execution time limit = PT2M
-     [[[directives]]]
-         --mem = 3G
+#. The recipe should now be added to your ``<site>-recipes.cylc`` file. (Where
+   ``<site>`` is your site). Find this in the
+   ``esmvaltool/utils/recipe_test_workflow/site/`` directory.
+
+#. ``<site>-recipes.cylc`` contains **two lists of dictionaries**. The lists
+   are ``FAST_RECIPES`` and ``MEDIUM_RECIPES``.
 
    .. hint::
-      The ``fast`` key in the example task definition above
-      (``[[process<fast=recipe_albedolandcover>]]``) should match name of the
-      option the recipe was added to in the ``[task parameters]`` section in
-      the ``esmvaltool/utils/recipe_test_workflow/flow.cylc`` file
+      ``FAST_RECIPES`` take *less* than 10 minutes to run at your site.
+      ``MEDIUM_RECIPES`` take *more* than 10 minutes.
+
+#. Add the recipe to one of lists as as a dictionary of ``key, value`` pairs.
+   E.g.::
+
+      {
+         'recipe_path': 'recipe_a_fast_recipe',
+         'actual': '1m30s, 1.5 GB on 2025-01-01',
+         'max_time': 'PT2M',
+         'max_memory': '2G',
+      }
+
+   Add the following information for each key:
+
+   .. list-table::
+      :widths: 25 75
+      :header-rows: 1
+
+      * - Key
+        - Value
+      * - ``recipe_path``
+        - The path to the recipe. Recipe paths are specified relative to
+          ``esmvaltool/recipes``. For recipes in subdirectories, ``--`` stands
+          for ``/`` since the latter is an illegal char.
+      * - ``actual``
+        - A note of  the recipe's actual resource usage. From the successful run
+          of the recipe on the compute server at your site.
+      * - ``max_time``
+        - Set the maximum amount of time the recipe has to complete. Use the
+          ISO8601 duration format (see `Cylc ISO8601 Durations`_ for more).
+      * - ``max_memory``
+        - Set the memory to allocate to running the recipe. Default units are
+          megabytes. Different units can be specified using the
+          suffix [K|M|G|T] (see `Slurm sbatch --mem`_ for more).
 
    .. hint::
-      Set the ``execution time limit`` to 10-20% more than the actual duration.
-      For actual durations of up to ``1m45s``, set the ``execution time limit``
-      to ``PT2M`` (2 minutes).
+      Set the ``max_time`` to 10-20% more than the actual duration. For actual
+      durations of up to ``1m45s``, set ``max_time`` to ``PT2M`` (2 minutes).
 
    .. hint::
       Try not to regularly waste more than 500 MiB in memory usage. Typically,
       rounding the actual memory usage up to the nearest integer is acceptable.
 
+#. Once the recipe dictionary is added to either ``FAST_RECIPES`` or
+   ``MEDIUM_RECIPES``, the recipe will run as part of the |RTW| at your site.
+   Next, the recipe will need |KGOs| to run successfully in the RTW.
+
+   .. _jinja2_templating_language:
+
+   .. note::
+      The ``<site>-recipes.cylc`` file is actually written in the `Jinja2`_
+      templating language. Jinja2 gives |Cylc| many powerful features (see
+      `Cylc Jinja2`_). This is beyond the scope of this guide. Follow the links
+      for more information.
+
+Update the KGOs
+---------------
+
 #. Stop any running ``recipe_test_workflow`` workflows::
 
     cylc stop recipe_test_workflow/*
@@ -104,6 +134,9 @@ How to add a recipe to the |RTW|
 #. Run the |RTW| again, as detailed in the :ref:`quick_start_guide`; the
    ``compare`` task should now succeed.
 
+Add the recipe to the documentation
+------------------------------------
+
 #. Add the recipe to the documentation:
 
    * Add a URL for the recipe in ``doc/sphinx/source/utils/RTW/common.txt``

doc/sphinx/source/utils/RTW/common.txt

@@ -14,20 +14,16 @@
 
 .. Links
 
-.. _Autoassess landsurface soilmoisture: https://docs.esmvaltool.org/en/latest/recipes/recipe_autoassess_landsurface_soilmoisture.html#available-recipes-and-diagnostics
-.. _Consecutive dry days: https://docs.esmvaltool.org/en/latest/recipes/recipe_consecdrydays.html
 .. _CMIP Documentation: https://www.wcrp-climate.org/wgcm-cmip
 .. _Cylc Documentation: https://cylc.github.io/cylc-doc/stable/html/index.html
+.. _Cylc ISO8601 Durations: https://cylc.github.io/cylc-doc/stable/html/tutorial/scheduling/datetime-cycling.html#iso8601-durations
+.. _Cylc Jinja2: https://cylc.github.io/cylc-doc/stable/html/user-guide/writing-workflows/jinja2.html
 .. _Cylc Review: https://cylc.github.io/cylc-doc/7.9.3/html/tutorial.html#viewing-suite-logs-in-a-web-browser-cylc-review
-.. _Ensclus: https://docs.esmvaltool.org/en/latest/recipes/recipe_ensclus.html
 .. _ESMValTool Documentation: https://esmvaltool.org/
 .. _ESMValTool issue #3663: https://github.com/ESMValGroup/ESMValTool/issues/3663
 .. _ESMValTool PR #3664: https://github.com/ESMValGroup/ESMValTool/pull/3664
-.. _Examples python: https://docs.esmvaltool.org/en/latest/recipes/recipe_examples.html
-.. _Heatwaves coldwaves: https://docs.esmvaltool.org/en/latest/recipes/recipe_heatwaves_coldwaves.html
-.. _Landcover - Albedo: https://docs.esmvaltool.org/en/latest/recipes/recipe_albedolandcover.html#landcover-albedo
+.. _Jinja2: https://jinja.palletsprojects.com/en/stable/
 .. _new ESMValTool issue: https://github.com/ESMValGroup/ESMValTool/issues/new
-.. _Ocean Atlantic Meridional Overturning Circulation (AMOC): https://docs.esmvaltool.org/en/latest/recipes/recipe_oceans.html#recipe-ocean-amoc-yml
-.. _Ocean multimap: https://docs.esmvaltool.org/en/latest/recipes/recipe_oceans.html#recipe-ocean-multimap-yml
-.. _Radiation budget: https://docs.esmvaltool.org/en/latest/recipes/recipe_radiation_budget.html
 .. _Rose Documentation: https://metomi.github.io/rose/doc/html/index.html
+.. _metoffice-recipes.cylc: https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/utils/recipe_test_workflow/site/metoffice-recipes.cylc
+.. _Slurm sbatch --mem: https://slurm.schedmd.com/sbatch.html#OPT_mem

doc/sphinx/source/utils/RTW/tested_recipes.rst

@@ -6,14 +6,16 @@ Currently tested recipes
 
 .. include:: common.txt
 
-The following currently tested recipes are available in the |RTW|:
+The recipes tested with the |RTW| at each site:
 
-* `Autoassess landsurface soilmoisture`_
-* `Consecutive dry days`_
-* `Ensclus`_
-* `Examples python`_
-* `Heatwaves coldwaves`_
-* `Landcover - Albedo`_
-* `Ocean Atlantic Meridional Overturning Circulation (AMOC)`_
-* `Ocean multimap`_
-* `Radiation budget`_
+* `metoffice-recipes.cylc`_
+* dkrz-recipes.cylc **(COMING SOON!)**
+* jasmin-recipes.cylc **(COMING SOON!)**
+
+.. hint::
+
+    - Information about the contents of these files:
+      :ref:`Run a Recipe in the RTW <run_a_recipe_in_the_rtw>`.
+    - The files use the Jinja2 templating language (see this
+      :ref:`note <jinja2_templating_language>`, the `Cylc Jinja2`_ documentation
+      or the `Jinja2`_ documentation for more information).

esmvaltool/utils/recipe_test_workflow/flow.cylc

@@ -1,21 +1,24 @@
 #!jinja2
+{{ assert(SITE != "", "SITE must be set to something other than an empty string") }}
+
+# Add recipes to the RTW in your site's `<site>-recipes.cylc` file in the 'site'
+# directory. See "How to add a recipe to the RTW" documentation for more information.
+{% set SITE_RECIPES_FILE = 'site/' ~ SITE ~ '-recipes.cylc' %}
+{% from SITE_RECIPES_FILE import FAST_RECIPES, MEDIUM_RECIPES %}
+
+{% set ERROR_MSG =
+    "is undefined and must be set, usually in '" ~ SITE_RECIPES_FILE ~ " - see 'How to "
+    "add a recipe to the RTW' documentation for more information."
+%}
+{{ assert(FAST_RECIPES is defined, "FAST_RECIPES " ~ ERROR_MSG)}}
+{{ assert(MEDIUM_RECIPES is defined, "MEDIUM_RECIPES " ~ ERROR_MSG)}}
+
 [scheduler]
     UTC mode = True
 
 [task parameters]
-    # See "How to add a recipe to the RTW" documentation for more information.
-    # Recipe paths are specified relative to esmvaltool/recipes. For recipes in
-    # subdirectories, `--` stands for `/` since the latter is an illegal char.
-    fast = recipe_radiation_budget, \
-        recipe_albedolandcover, \
-        recipe_heatwaves_coldwaves, \
-        recipe_autoassess_landsurface_soilmoisture, \
-        recipe_consecdrydays, \
-        examples--recipe_python, \
-        recipe_ocean_multimap, \
-        recipe_ocean_amoc
-    medium = recipe_ensclus  # Should be in fast: is in medium temporarily to
-                             # avoid empty variable.
+    fast = {{ FAST_RECIPES | map(attribute='recipe_path') | join(', ') }}
+    medium = {{ MEDIUM_RECIPES | map(attribute='recipe_path') | join(', ') }}
 
 [scheduling]
     initial cycle point = now
@@ -78,17 +81,27 @@
 
     # Bash variable substitution below replaces `--` with `/` to obtain path to
     # recipe.
-    [[process<fast>]]
+    {% for RECIPE in FAST_RECIPES %}
+    [[process<fast={{ RECIPE['recipe_path'] }}>]]
         inherit = None, COMPUTE
+        execution time limit = {{ RECIPE['max_time'] }}
         [[[environment]]]
             ROSE_TASK_APP = process
             RECIPE_PATH = "${CYLC_TASK_PARAM_fast//--//}.yml"
+        [[[directives]]]
+            --mem = {{ RECIPE['max_memory'] }}
+    {% endfor %}
 
-    [[process<medium>]]
+    {% for RECIPE in MEDIUM_RECIPES %}
+    [[process<medium={{ RECIPE['recipe_path'] }}>]]
         inherit = None, COMPUTE
+        execution time limit = {{ RECIPE['max_time'] }}
         [[[environment]]]
             ROSE_TASK_APP = process
             RECIPE_PATH = "${CYLC_TASK_PARAM_medium//--//}.yml"
+        [[[directives]]]
+            --mem = {{ RECIPE['max_memory'] }}
+    {% endfor %}
 
     [[COMPARE]]
         inherit = None, COMPUTE

esmvaltool/utils/recipe_test_workflow/site/metoffice-recipes.cylc

@@ -0,0 +1,76 @@
+#   Add recipes to the RTW at your site by adding the required key, value pairs below.
+#   See "How to add a recipe to the RTW" documentation for more information.
+#
+#   KEYS                VALUES
+#   ----                ------
+#   recipe_path         Recipe paths are specified relative to esmvaltool/recipes. For
+#                       recipes in subdirectories, `--` stands for `/` since the latter
+#                       is an illegal char.
+#   actual              Example recorded usage at your site.
+#   max_time            The maximum amount of time the recipe has to complete.
+#   max_memory          The memory to allocate to running the recipe.
+
+{% set FAST_RECIPES =
+    [
+        {
+            'recipe_path': 'recipe_radiation_budget',
+            'actual': 'na',
+            'max_time': 'PT2M',
+            'max_memory': '2G',
+        },
+        {
+            'recipe_path': 'recipe_albedolandcover',
+            'actual': '0m31s, 2.5 GB on 2024-04-08',
+            'max_time': 'PT2M',
+            'max_memory': '3GB',
+        },
+        {
+            'recipe_path': 'recipe_heatwaves_coldwaves',
+            'actual': 'na',
+            'max_time': 'PT2M',
+            'max_memory': '2G',
+        },
+        {
+            'recipe_path': 'recipe_autoassess_landsurface_soilmoisture',
+            'actual': '0m36s, 2.8 GB on 2024-03-29',
+            'max_time': 'PT2M',
+            'max_memory': '3G',
+        },
+        {
+            'recipe_path': 'recipe_consecdrydays',
+            'actual': 'na',
+            'max_time': 'PT2M',
+            'max_memory': '2G',
+        },
+        {
+            'recipe_path': 'examples--recipe_python',
+            'actual': 'na',
+            'max_time': 'PT2M',
+            'max_memory': '2G',
+        },
+        {
+            'recipe_path': 'recipe_ocean_amoc',
+            'actual': '1m47s, 3.4 GB on 2024-06-17',
+            'max_time': 'PT3M',
+            'max_memory': '4G',
+        },
+        {
+            'recipe_path': 'recipe_ocean_multimap',
+            'actual': '6m52s, 2.5 GB on 2024-05-28',
+            'max_time': 'PT8M',
+            'max_memory': '3G',
+        },
+    ]
+%}
+
+# `recipe_ensclus` should be in fast: is in medium temporarily to avoid empty variable.
+{% set MEDIUM_RECIPES =
+    [
+        {
+            'recipe_path': 'recipe_ensclus',
+            'actual': '2m10s, 2.4 GB on 2024-03-29',
+            'max_time': 'PT4M',
+            'max_memory': '3G',
+        },
+    ]
+%}

esmvaltool/utils/recipe_test_workflow/site/metoffice.cylc

@@ -1,4 +1,5 @@
 #!jinja2
+
 [runtime]
     [[root]]
         [[[environment]]]
@@ -7,14 +8,11 @@
             PYTHONPATH_PREPEND = ${ESMVALCORE_DIR}:${ESMVALTOOL_DIR}
 
     # COMPUTE provides defaults for computation-heavy tasks.
-    # Specific tasks below override some defaults, e.g. time & memory.
     [[COMPUTE]]
         platform = spice
-        execution time limit = PT2M
         [[[directives]]]
             --wckey = RTW
             --ntasks = {{ MAX_PARALLEL_TASKS }}
-            --mem = 2G
 
     [[configure]]
         platform = localhost
@@ -24,37 +22,3 @@
         execution time limit = PT2M
         [[[directives]]]
             --mem = 2G
-
-    # Resources for recipes that need more than the default. Both time and
-    # memory should be specified, in case the default changes.
-    # Variable (fast, medium) must be consistent with flow.cylc.
-    # Comment should indicate example recorded usage at Met Office.
-    [[process<fast=recipe_autoassess_landsurface_soilmoisture>]]
-        # Actual: 0m36s, 2.8 GB on 2024-03-29.
-        execution time limit = PT2M
-        [[[directives]]]
-            --mem = 3G
-
-    [[process<fast=recipe_albedolandcover>]]
-        # Actual: 0m31s, 2.5 GB on 2024-04-08.
-        execution time limit = PT2M
-        [[[directives]]]
-            --mem = 3G
-
-    [[process<fast=recipe_ocean_amoc>]]
-        # Actual: 1m47s, 3.4 GB on 2024-06-17.
-        execution time limit = PT3M
-        [[[directives]]]
-            --mem = 4G
-
-    [[process<fast=recipe_ocean_multimap>]]
-        # Actual: 6m52s, 2.5 GB on 2024-05-28.
-        execution time limit = PT8M
-        [[[directives]]]
-            --mem = 3G
-
-    [[process<medium=recipe_ensclus>]]
-        # Actual: 2m10s, 2.4 GB on 2024-03-29.
-        execution time limit = PT4M
-        [[[directives]]]
-            --mem = 3G