Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MUON: add remote workflows for fwd matched tracks #638

Merged
merged 2 commits into from
Mar 18, 2024
Merged

MUON: add remote workflows for fwd matched tracks #638

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4723ed6
[MUON] added remote workflow for fwd matched tracks
aferrero2707 Mar 14, 2024
e4409f3
[GLO] removed fairmq-shmmonitor from qc-remote-workflow-glo and qc-re…
aferrero2707 Mar 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions jit/glo-mchmid-mtch-qcmn-remote
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
o2-qc --config consul-json://{{ consul_endpoint }}/o2/components/qc/ANY/any/glo-mchmid-mtch-qcmn-epn --remote -b
1 change: 1 addition & 0 deletions jit/glo-mftmch-mtch-qcmn-remote
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
o2-qc --config consul-json://{{ consul_endpoint }}/o2/components/qc/ANY/any/glo-mftmch-mtch-qcmn-epn --remote -b
1 change: 1 addition & 0 deletions jit/glo-mftmchmid-mtch-qcmn-remote
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
o2-qc --config consul-json://{{ consul_endpoint }}/o2/components/qc/ANY/any/glo-mftmchmid-mtch-qcmn-epn --remote -b
49 changes: 41 additions & 8 deletions workflows/readout-dataflow.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -891,14 +891,23 @@ defaults:
panel: "QC_Nodes_Workflows"
index: 21
visibleif: $$qc_remote_enabled === "true"
glo_fwd_mtch_qc_enabled: !public
value: "false"
type: bool
label: "GLO Forward Matching QC"
description: "Enable/disable QC node workflow for forward Matching QC. Deployed only if MCH present!"
widget: checkBox
panel: "QC_Nodes_Workflows"
index: 22
visibleif: $$qc_remote_enabled === "true"
glo_mean_vtx_calib_post_proc_qc_enabled: !public
value: "false"
type: bool
label: "GLO Mean Vertex calib post-processing QC"
description: "Enable/disable QC node workflow for processing the Mean Vertex calibration objects. Deployed only if ITS present!"
widget: checkBox
panel: "QC_Nodes_Workflows"
index: 22
index: 23
visibleif: $$qc_remote_enabled === "true"
glo_big_screen_qc_enabled: !public
value: "false"
Expand All @@ -907,7 +916,7 @@ defaults:
description: "Enable/disable QC Big Screen"
widget: checkBox
panel: "QC_Nodes_Workflows"
index: 23
index: 24
visibleif: $$qc_remote_enabled === "true"
###############################
# TRG Panel
Expand Down Expand Up @@ -1774,13 +1783,37 @@ roles:
- name: glo-qc-big-screen
enabled: "{{ glo_big_screen_qc_enabled == 'true' && qc_remote_jit_enabled == 'true'}}"
include: "{{ qc_remote_jit_enabled == 'true' ? dpl.GenerateFromUri('glo-qc-big-screen') : '' }}"
- name: fairmq-shmmonitor
enabled: "{{ fmq_cleanup_enabled == 'true' && qc_remote_workflow != 'none' }}"
task:
load: "fairmq-shmmonitor"
trigger: DESTROY
timeout: 10s
# When qc_remote_enabled is true, it will try to deploy a qc_remote_workflow for every detector in the list.
# However if the {{detector}}_qc_remote_workflow is none, we deploy a ghost role that does nothing to preserve
# control tree state management.
- name: dpl-noop-subwf
enabled: "{{ qc_remote_workflow == 'none' }}"
call:
func: testplugin.Noop()
trigger: CONFIGURE
timeout: 1s
critical: false
- name: qc-remote-workflow-fwd # GLO is not a detector so we won't iterate on it with 'detectors' and we need special enable logic
enabled: "{{ 'MCH' in json.Unmarshal(detectors) }}"
vars:
qc_remote_workflow: "{{ util.PrefixedOverride( 'qc_remote_workflow', 'glo' ) }}"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@knopers8 it is ok to use the glo prefix for the workflow in two separate rules?

Copy link
Collaborator

@knopers8 knopers8 Mar 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, i don't think this variable is used for generating the actual workflow. It is probably a leftover of earlier approaches.

The only potential issue I can see is that qc_remote_workflow != 'none' is used to enable shmem cleanup. While I see in a prod environment that it is not enabled, to be on the safe side, I would propose to remove the fairmq-shmmonitor from this role and from the role qc-remote-workflow-glo.

The problem with this could appear if we have two environments, one for ITS, one for MCH, running at the same time. The first one to be destroyed will affect the rest of the processing of the other environment on the same machine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@knopers8 thanks! I have removed the fairmq-shmmonitor from the two roles. I will organize a test in production with replay data to check that all is fine, and post here the outcome. The initial version was already checked in staging.

qc_dpl_command: "{{ util.PrefixedOverride( 'qc_dpl_command', 'glo' ) }}"
qc_remote_machine: "{{ util.PrefixedOverride( 'qc_remote_machine', 'glo' ) }}"
shm_segment_size: "{{ util.PrefixedOverride( 'qc_shm_segment_size', 'glo' ) }}"
detector: GLO
constraints:
- attribute: machine_id
value: "{{ qc_remote_machine }}"
roles:
- name: glo-fwd-mchmid
enabled: "{{ glo_fwd_mtch_qc_enabled == 'true' && qc_remote_jit_enabled == 'true' && 'MFT' in json.Unmarshal(detectors) && !('MID' in json.Unmarshal(detectors)) }}"
include: "{{ qc_remote_jit_enabled == 'true' ? dpl.GenerateFromUri('glo-mftmch-mtch-qcmn-remote') : '' }}"
- name: glo-fwd-mftmch
enabled: "{{ glo_fwd_mtch_qc_enabled == 'true' && qc_remote_jit_enabled == 'true' && 'MID' in json.Unmarshal(detectors) && !('MFT' in json.Unmarshal(detectors)) }}"
include: "{{ qc_remote_jit_enabled == 'true' ? dpl.GenerateFromUri('glo-mchmid-mtch-qcmn-remote') : '' }}"
- name: glo-fwd-mftmchmid
enabled: "{{ glo_fwd_mtch_qc_enabled == 'true' && qc_remote_jit_enabled == 'true' && 'MFT' in json.Unmarshal(detectors) && 'MID' in json.Unmarshal(detectors) }}"
include: "{{ qc_remote_jit_enabled == 'true' ? dpl.GenerateFromUri('glo-mftmchmid-mtch-qcmn-remote') : '' }}"
# When qc_remote_enabled is true, it will try to deploy a qc_remote_workflow for every detector in the list.
# However if the {{detector}}_qc_remote_workflow is none, we deploy a ghost role that does nothing to preserve
# control tree state management.
Expand Down
Loading