Added example specification files #4

eacharles · 2023-10-12T22:20:40Z

examples/config_dc2_test_med.yaml is the top-level specification file and it calls out all the other ones.

examples/notes.txt shows some thoughts on how we might use these to build and run campaigns

ctslater

Added a few comments on naming and clarifying the schema. Functionally it looks good.

ctslater · 2023-10-13T16:28:36Z

examples/notes.txt

+# Initial setup of the campaign template.  This only happen once per campaign type
+
+# Create a production as a namespace
+cm-service add production --name dc2_test_med


I think the user interacts with something like cm-client, and the service is something running elsewhere.

Good point, we should change the name of the client-using cli accordingly.

ctslater · 2023-10-13T16:56:36Z

examples/notes.txt

+cm-service add production --name dc2_test_med
+
+# Load the related specification.
+cm-service load specification --yaml_file examples/config_dc2_test_med.yaml --production_name dc2_test_med --spec_name v0 --set-as-default


I also think we need a name for the thing referred to here as yaml_file; campaign config? production config? something template?

Good point. What we call it is going to depend a bit on the relationship between the block in the specification files, thing "glue" class that gathers then, and campaigns.

ctslater · 2023-10-13T17:00:48Z

examples/config_dc2_test_med.yaml

+              child_config:
+                  base_query: "instrument='LSSTCam-imSim' and skymap='DC2' and tract in (3828, 3829)"
+      # Campaign level paramters
+      data:


Could be more specific, maybe handler_parameters:? extra_handler_parameters? What is the relationship between this and child_config entries?

Sorry, do you mean the "data:" block? If so, they are similar, but the idea is that the "data" block contains things that are applicable all the way down the hierarchy, whereas the child_config is only applicable to the next level down.

And yes, 'parameters' or 'heritage' or 'parameters_passed_to_children' would be move precise.

ctslater · 2023-10-13T17:05:43Z

examples/config_standard_elements.yaml

+
+
+# Import the common script templates
+- import: "${CM_CONFIGS}/config_standard_scripts.yaml"


If there was a top-level dictionary element like CampaignTemplate, then it could have (dictionary representation just for simplicity):
CampaignTemplate: {"imports": [], "blocks": []}
That would make it so every list contained a consistent type of element.

I'm not sure I understand how that would work with multiple levels of imports, or would the idea be not to have multiple levels of imports.

I'm not sure what multiple levels of imports refers to.

file 1 imports file 2 which imports file 3...
hard to do that if the imports are defined in some sort of top-level

ctslater · 2023-11-06T23:28:06Z

examples/config_dc2_test_med.yaml

+          root: 'cm/dc2_test_med'
+      # Define the steps, their connections and override parameters as needed
+      child_config:
+          step1:


Is 'step1' here a reference or is it defining a name?

reference, it could be
- step:
spec_block: step1
otherstuff: stuff
- step:
spec_block: step2
...

ctslater · 2023-11-06T23:34:04Z

examples/config_dc2_test_med.yaml

+      child_config:
+          step1:
+              spec_block: dc2_step1
+              child_config:


I'm not sure I understand the schema of child_config; some of the child elements are whole steps and some are kwarg-like parameters? Those seem like different enough purposes that it might be clearer if they were separate dictionaries?

Yes, the hierarchy is pretty complicated, if you have a sense of how to split it that might be clearer, we could have a go.

eigerx · 2023-11-07T19:21:28Z

examples/bps_config_template.yaml

@@ -0,0 +1,9 @@
+includeConfigs:


Looks familiar!

eigerx · 2023-11-07T19:26:54Z

examples/manifest_script_template.sh

+# setup LSST env.
+export WEEKLY='{lsst_version}'
+source /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/${WEEKLY}/loadLSST.bash
+setup lsst_distrib


Does manifest checking happen in this script or somewhere else?

This is the wrapper that sets up the env for that script. That script gets defined by the ManfestReportScriptHandler, here:

cm-service/src/lsst/cmservice/handlers/jobs.py

Line 292 in 05a9a04

class ManifestReportScriptHandler(ScriptHandler):

eigerx · 2023-11-07T19:29:56Z

examples/config_dc2_steps.yaml

+      includes: ['step']
+      data:
+          pipeline_yaml: "${DRP_PIPE_DIR}/pipelines/LSSTCam-imSim/DRP-test-med-1.yaml#step1"
+      child_config:


I see here that the groups are defined as the child_config of the step, and all the group information is below.

eigerx · 2023-11-07T19:30:38Z

examples/config_dc2_steps.yaml

+      data:
+          pipeline_yaml: "${DRP_PIPE_DIR}/pipelines/LSSTCam-imSim/DRP-test-med-1.yaml#step1"
+      child_config:
+          spec_block: group


What is the difference between SpecBlock and spec_block ? I'm concerned about confusion in editing these configs.

SpecBlock defines a thing, spec_block refers that thing by name.

Would it perhaps be a little more clear to call spec_block something like parent_spec_block? Or am I misinterpreting the use of it?

parent is a little overloaded here in that it also refers to the parent node in the hierarchy. Here and elsewhere spec_block is refering to "the SpecBlock used to build that child".

eigerx · 2023-11-07T19:31:30Z

examples/config_dc2_steps.yaml

+- SpecBlock:
+      name: dc2_step1
+      includes: ['step']
+      data:


What kind of category is data here? I see pipeline_yaml but what other information might we want in data ?

Anything that gets passed down the heirarchy that isn't collection name templates (that goes in collections), configuration of how to build child nodes (that goes in child_config) or renaming what spec blocks to use (that goes in spec_aliases)

eigerx · 2023-11-07T19:40:03Z

examples/config_standard_elements.yaml

+                prerequisites: ['bps_report']
+                collections:
+                    run: "{job_run}"
+      data:


How does rescue: false also fall under data ?

We could chose to define a job that has rescue: true, which would use the same template, but just override that one parameter,

eigerx · 2023-11-07T19:45:13Z

examples/config_dc2_test_med.yaml

+              prerequisites: ['step1']
+              child_config:
+                  base_query: "instrument='LSSTCam-imSim' and skymap='DC2' and tract in (3828, 3829)"
+          step3:


This file all looks pretty familiar. I think it's decently straightforward. It's not clear what makes some things fall under child_config. I'm getting the impression that all the spec_block s here are linked config_dc2_steps.yaml which seems good. What does the linking/how does that work?

child_config is pretty much anything that you need to know how to make the child nodes.

the linking happens when you make the child nodes, as part of that process you look up which spec blocks to use as templates for each child node. You can see an example of that around here:

cm-service/src/lsst/cmservice/handlers/element_handler.py

Line 144 in 05a9a04

for script_item in spec_block.scripts:

eacharles · 2024-01-18T22:04:49Z

This has been overtaken by events and the comments have been incorporated into the design of the specification files.

fritzm force-pushed the tickets/DM-41174 branch 2 times, most recently from e24a21d to 39ce381 Compare October 23, 2023 02:44

fritzm force-pushed the tickets/DM-41174 branch from 39ce381 to 587628e Compare November 4, 2023 22:12

ctslater reviewed Nov 7, 2023

View reviewed changes

eigerx reviewed Nov 7, 2023

View reviewed changes

fritzm force-pushed the tickets/DM-41174 branch 2 times, most recently from 1f42d4b to 8e52f14 Compare November 15, 2023 02:29

fritzm force-pushed the tickets/DM-41174 branch from 8e52f14 to 37fe817 Compare November 29, 2023 10:19

Added example specification files

beff8c4

fritzm force-pushed the tickets/DM-41174 branch from 37fe817 to beff8c4 Compare December 4, 2023 03:17

eacharles closed this Jan 18, 2024

eacharles deleted the tickets/DM-41174 branch May 28, 2024 16:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added example specification files #4

Added example specification files #4

eacharles commented Oct 12, 2023

ctslater left a comment

ctslater Oct 13, 2023

eacharles Nov 7, 2023

ctslater Oct 13, 2023

eacharles Nov 7, 2023

ctslater Oct 13, 2023

eacharles Nov 7, 2023

eacharles Nov 7, 2023

ctslater Oct 13, 2023

eacharles Nov 7, 2023

ctslater Nov 7, 2023 •

edited

Loading

eacharles Nov 7, 2023

ctslater Nov 6, 2023

eacharles Nov 7, 2023

ctslater Nov 6, 2023

eacharles Nov 7, 2023

eigerx Nov 7, 2023

eigerx Nov 7, 2023

eacharles Nov 7, 2023

eigerx Nov 7, 2023

eigerx Nov 7, 2023

eacharles Nov 7, 2023

villarrealas Nov 7, 2023

eacharles Nov 7, 2023

eigerx Nov 7, 2023

eacharles Nov 7, 2023

eigerx Nov 7, 2023

eacharles Nov 7, 2023

eigerx Nov 7, 2023

eacharles Nov 7, 2023

eacharles commented Jan 18, 2024



		# Import the common script templates
		- import: "${CM_CONFIGS}/config_standard_scripts.yaml"

Added example specification files #4

Added example specification files #4

Conversation

eacharles commented Oct 12, 2023

ctslater left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ctslater Nov 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eacharles commented Jan 18, 2024

ctslater Nov 7, 2023 •

edited

Loading