Use site-specific lists of recipes in the RTW #3856

chrisbillowsMO · 2025-01-09T10:22:29Z

Description

Closes Use site-specific lists of recipes in the RTW #3724
Link to documentation: https://esmvaltool--3856.org.readthedocs.build/en/3856/
Link to RTW | How to Add A Recipe to the RTW

Before you get started

☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

🛠 This pull request has a descriptive title
🛠 Code is written according to the code quality guidelines
🛠 Documentation is available
🛠 Tests run successfully
🛠 All checks below this pull request were successful

To help with the number of pull requests:

🙏 We kindly ask you to review two other open pull requests in this repository

ehogan

Great work @chrisbillowsMO 🥳

I haven't yet reviewed the documentation, but I have run out of time today, so I'm sharing the review comments I have made so far. It's unlikely I'll have any time tomorrow to complete the review, so I'll do it first thing on Monday 👍

doc/sphinx/source/utils/RTW/tested_recipes.rst

ehogan · 2025-01-09T18:15:23Z

doc/sphinx/source/utils/RTW/tested_recipes.rst

+    - The files use the Jinja2 templating language (see this
+      :ref:`note <jinja2_templating_language>`, the `Cylc Jinja2`_ documentation
+      or the `Jinja2`_ documentation for more information).


Adding a link to more infomation (the first link) is great! 🥳 I wonder whether this second link should be included somewhere within the first link (if that's where more information about the files is located)? 🤔

The second and third links were in the first link AS WELL...! Which is how I prefer it - but I merrily defer to the single source of truth doctrine. Despicable duplicate links removed here: 3445729 😉

ehogan · 2025-01-09T18:23:02Z

esmvaltool/utils/recipe_test_workflow/flow.cylc

+    "is undefined and must be set, usually in '" ~ SITE_RECIPES_FILE ~ " - see 'How to "
+    "add a recipe to the RTW' documentation for more information."


Clarify the request. Also, I found out recently that, when writing documentation: use inclusive language that doesn't always refer to "seeing" an element. Unfortunately this is something I used to do a lot and I keep finding occurrences of it! 😱 I have made a suggestion below, but feel free to adjust 😊

Suggested change

"is undefined and must be set, usually in '" ~ SITE_RECIPES_FILE ~ " - see 'How to "

"add a recipe to the RTW' documentation for more information."

"is undefined and must be set in '" ~ SITE_RECIPES_FILE ~ ". The 'How to "

"add a recipe to the RTW' documentation provides more information."

I commited your suggestion but have reverted the . to a - (keeping your text) here: 2bb79fc

Jinja2 renders . as a line break.
E.g.

Jinja2Error: Jinja2 Assertion Error: FAST_RECIPES is undefined and must be set in 'site/metoffice-recipes.cylc'. The 'How to add a recipe to the RTW' documentation provides more information.

I did experiment with getting a . (e.g. using \., {% raw %} from here, using an utf-8 full stop) - but it made me feel like too much of a failure. I could live with the dash...!

(I didn't comment it in the file; it didn't feel a big enough issue. Let me know if we should have a comment)

ehogan · 2025-01-09T18:23:55Z

esmvaltool/utils/recipe_test_workflow/flow.cylc

+# Add recipes to the RTW in your site's `<site>-recipes.cylc` file in the 'site'
+# directory. See "How to add a recipe to the RTW" documentation for more information.


Would it be possible to remove this comment? It's almost identical to the error message below 😊

I just wanted them to REALLY know! 🤣

Removed here: 6054822

ehogan · 2025-01-09T18:26:35Z

esmvaltool/utils/recipe_test_workflow/flow.cylc

@@ -78,17 +81,27 @@

    # Bash variable substitution below replaces `--` with `/` to obtain path to
    # recipe.
-    [[process<fast>]]
+    {% for RECIPE in FAST_RECIPES %}


Jinja2 code should be indented from the left margin on its own terms, as recommended in Cylc Documentation: Style Guidelines

Done 79ac7af (but yuck! 😆)

ehogan · 2025-01-09T18:33:08Z

esmvaltool/utils/recipe_test_workflow/site/metoffice-recipes.cylc

+    [
+        {
+            'recipe_path': 'recipe_radiation_budget',
+            'actual': 'na',


Would it be possible for you to get these values from the logs from the latest successful run (using your environment), please? 😊

Added here: 9c65801 I increased memory for two recipes based on the actuals - per the docs. They've obviously been running fine but I thought, if we're the "example" (at least for now) consistency with the documentation was most important.

And I alphabetised the recipes here: 6a61034

esmvaltool/utils/recipe_test_workflow/site/metoffice-recipes.cylc

ehogan · 2025-01-09T18:36:18Z

esmvaltool/utils/recipe_test_workflow/site/metoffice-recipes.cylc

+{% set FAST_RECIPES =
+    [
+        {
+            'recipe_path': 'recipe_radiation_budget',


Are single quotes required, or can double quotes be used? 🤔

Replaced all singles with doubles: 7d7f19c

ehogan · 2025-01-09T18:37:56Z

esmvaltool/utils/recipe_test_workflow/site/metoffice-recipes.cylc

+# `recipe_ensclus` should be in fast: is in medium temporarily to avoid empty variable.
+{% set MEDIUM_RECIPES =
+    [
+        {
+            'recipe_path': 'recipe_ensclus',
+            'actual': '2m10s, 2.4 GB on 2024-03-29',
+            'max_time': 'PT4M',
+            'max_memory': '3G',
+        },
+    ]
+%}


Now you have added checks (thank you!), if this was moved to FAST_RECIPES (where it belongs), would the workflow run without errors? 🤔

Unfortunately, I don't think so. If MEDIUM_RECIPES is empty (list or string) we still get:

ParamExpandError: parameter medium is not defined in compare<medium>

esmvaltool/utils/recipe_test_workflow/site/metoffice.cylc

ehogan · 2025-01-09T19:01:11Z

esmvaltool/utils/recipe_test_workflow/flow.cylc

+{% set SITE_RECIPES_FILE = 'site/' ~ SITE ~ '-recipes.cylc' %}
+{% from SITE_RECIPES_FILE import FAST_RECIPES, MEDIUM_RECIPES %}


I know you spent a lot of time investigating this, and apologies if you already told me the answer to this question, but if the file is included (e.g. {% include 'site/' ~ SITE ~ '-recipes.cylc' %}) are the FAST_RECIPES and MEDIUM_RECIPES variables not recognised? 🤔

Co-authored-by: Emma Hogan <[email protected]>

chrisbillowsMO added 15 commits December 17, 2024 15:08

#3724: Use temp environment

84c6a66

#3724: Add Jinja2 exclude list. Not working for all recipes

e803843

#3724: Add comments to explain when Jinja2 would error

de0b3bf

#3724: Working prototype of import from inc solution

bed1f75

#3724: Add j2 errors for unset SITE, FAST/MEDIUM_RECIPES

db1a048

Merge branch 'main' into 3724_rtw_site_specific_recipes

0b3971e

#3724: Add unsatisfactory j2 error for missing custom directives recipe

6c2f04b

#3724: Move custom directives to site recipes.cylc

50fe052

#3724: Fix cylc linting

9127811

#3724: Move metoffice-recipes.cylc to site

7421e67

#3724: Remove recipe defaults. Add settings to individual recipes

b785a76

#3724: Move the now generic parameterisation into flow.cylc

913854c

#3724: Remove reference to now deleted default settings

f93bee9

#3724: Update docs and module docstring

1ae2940

#3724: Tweaks docs

cc0cd2f

chrisbillowsMO added the Recipe Test Workflow (RTW) Items relevant to the Recipe Test Workflow label Jan 9, 2025

chrisbillowsMO self-assigned this Jan 9, 2025

chrisbillowsMO added 3 commits January 9, 2025 10:27

#3724: Remove temp env. Restore community env

c3bf56c

#3724: Doc tweak and adjust line lengths to < 80

f3a73c6

Merge branch 'main' into 3724_rtw_site_specific_recipes

5a876a5

chrisbillowsMO added this to the v2.12.0 milestone Jan 9, 2025

#3724: Doc string correction

0994bc9

chrisbillowsMO marked this pull request as ready for review January 9, 2025 11:21

chrisbillowsMO requested review from alistairsellar and ehogan as code owners January 9, 2025 11:21

chrisbillowsMO requested review from ehogan and removed request for ehogan January 9, 2025 11:21

chrisbillowsMO added 2 commits January 9, 2025 11:22

Merge branch 'main' into 3724_rtw_site_specific_recipes

64664c6

#3724: Revise "Currently tested recipes" page

0ab631c

chrisbillowsMO mentioned this pull request Jan 9, 2025

Update DKRZ Recipe Test Workflow to the use <site>-recipes.cylc #3857

Open

1 task

chrisbillowsMO mentioned this pull request Jan 9, 2025

Update JASMIN Recipe Test Workflow to the use <site>-recipes.cylc #3858

Open

1 task

ehogan requested changes Jan 9, 2025

View reviewed changes

ehogan reviewed Jan 9, 2025

View reviewed changes

chrisbillowsMO and others added 14 commits January 10, 2025 09:36

#3724: Rephrase doc and docstring

73232b0

Co-authored-by: Emma Hogan <[email protected]>

#3724: Make module docstring site specific

6e3553c

Co-authored-by: Emma Hogan <[email protected]>

#3724: Correct 'GB' to 'G'

d489c22

Co-authored-by: Emma Hogan <[email protected]>

#3724: Add back empty line

3983a7b

Co-authored-by: Emma Hogan <[email protected]>

#3724: Remove extraneous comment

6054822

#3724: Unindent jinja2

79ac7af

#3724: Replace single quotes with double quotes

7d7f19c

#3724: Use a - to avoid jinja2 rendering a line break

2bb79fc

#3724: Add actual resources for former default recipes

9c65801

#3724: Alphabetise recipes

6a61034

#3724: Remove duplicate links

3445729

#3724: Update references to how tested recipes are documented

c484c89

#3724: More inclusive language

1713154

Merge branch 'main' into 3724_rtw_site_specific_recipes

477013d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use site-specific lists of recipes in the RTW #3856

Use site-specific lists of recipes in the RTW #3856

chrisbillowsMO commented Jan 9, 2025 •

edited

Loading

ehogan left a comment

ehogan Jan 9, 2025

chrisbillowsMO Jan 10, 2025 •

edited

Loading

ehogan Jan 9, 2025

chrisbillowsMO Jan 10, 2025

ehogan Jan 9, 2025

chrisbillowsMO Jan 10, 2025

ehogan Jan 9, 2025

chrisbillowsMO Jan 10, 2025

ehogan Jan 9, 2025

chrisbillowsMO Jan 10, 2025

ehogan Jan 9, 2025

chrisbillowsMO Jan 10, 2025

ehogan Jan 9, 2025

chrisbillowsMO Jan 10, 2025

ehogan Jan 9, 2025

		"is undefined and must be set, usually in '" ~ SITE_RECIPES_FILE ~ " - see 'How to "
		"add a recipe to the RTW' documentation for more information."

		# Add recipes to the RTW in your site's `<site>-recipes.cylc` file in the 'site'
		# directory. See "How to add a recipe to the RTW" documentation for more information.

		{% set SITE_RECIPES_FILE = 'site/' ~ SITE ~ '-recipes.cylc' %}
		{% from SITE_RECIPES_FILE import FAST_RECIPES, MEDIUM_RECIPES %}

Use site-specific lists of recipes in the RTW #3856

Are you sure you want to change the base?

Use site-specific lists of recipes in the RTW #3856

Conversation

chrisbillowsMO commented Jan 9, 2025 • edited Loading

Description

Before you get started

Checklist

ehogan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrisbillowsMO Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrisbillowsMO commented Jan 9, 2025 •

edited

Loading

chrisbillowsMO Jan 10, 2025 •

edited

Loading