Include ZDC in L1T re-emulation workflows

[missirol]

Weeks ago, the L1-uGT emulator (#42634) and unpacker (#42733) were updated to make use of data from the Zero-Degree Calorimeter, for the 2023 HIon run (and beyond). These updates were the necessary ones to be able to correctly take data online with a L1T menu including algorithms using ZDC data.

Other updates needed to properly test/re-emulate such L1T menus offline are still missing. For what I understand, this includes (at least) the following.

• A "packer" ("digi to raw") of the ZDC digis, as part of L1TDigiToRaw.
• The necessary updates to include this in all the relevant sequences/workflows to re-emulate the L1T results, e.g. SimL1EmulatorRepack_Full_cff.

As long as these updates are missing, any re-emulation of the L1T results in standard workflows will return incorrect results for any L1T algorithms using ZDC (again, for what I understand).

FYI: @cms-sw/hlt-l2

[cmsbuild]

A new Issue was created by @missirol Marino Missiroli.

@Dr15Jones, @sextonkennedy, @smuzaffar, @rappoccio, @antoniovilela, @makortel can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[mmusich]

assign l1

[cmsbuild]

New categories assigned: l1

@epalencia,@aloeliger you have been requested to review this Pull request/Issue and eventually sign? Thanks

[aloeliger]

@missirol @mmusich, I spoke with the ZDC developers a few weeks ago about providing a packer for their workflows. No update since then.

[mmusich]

, I spoke with the ZDC developers a few weeks ago about providing a packer for their workflows. No update since then.

Might be useful to tag them here as well if you know the gutHub accounts.
I don't know what are the general CMS plans for HI MC production, but looks like the ZDC L1T packer would be a necessary ingredient for the HLT simulation.

[aloeliger]

@matt2275 Just tagging you on this. @cfmcginn We didn't talk about this, but you developed the original ZDC emulator, so perhaps you should be in the loop.

[matt2275]

I have a first draft of a ZDC packer here (https://github.com/matt2275/cmssw/tree/ZDCUnpacker/EventFilter/L1TRawToDigi). I have not done any testing since I'm not sure how to do that. One way of testing this would be to produce two sets of digis one that was only unpacked once and the other that was unpacked, repacked and unpacked again. If the sets are identical, it's a good sign the packer is working properly. Unfortunately, I'm not sure how to go about doing that.

[missirol]

@cms-sw/l1-l2

What's the status of this issue ?

[mmusich]

What's the status of this issue?
@cms-sw/l1-l2 @matt2275

[mmusich]

IIUC

A "packer" ("digi to raw") of the ZDC digis, as part of L1TDigiToRaw.

was done at #44019

What about the item

The necessary updates to include this in all the relevant sequences/workflows to re-emulate the L1T results, e.g. SimL1EmulatorRepack_Full_cff.

[hjbossi]

Hi @mmusich, the L1TZDC emulation is included here: #42818 - is this what you mean or something else? Of course, this is also dependent on the outstanding issue related to the geometry (see #43582) but this is being fixed by @bsunanda .

[hjbossi]

tagging also @Michael-Krohn, @cfmcginn, @abdoulline

[matt2275]

@mmusich the packer was added with #44019.

I do not about the status of updating the relevant sequences and workflows

[abdoulline]

I hope @Michael-Krohn can correct me, but my understanding of the caveat with ZDC TP emulation ( #42818 ) is the following:

• it works/serves solely for data re-emulation for consistency checks of emulated vs on-detector ZDC TPs

Because:

• existing ZDC DB conditions correspond to Run1 legacy ZDC, as Run3 ZDC Geometry (-> channels serialization) is not yet available
• Run3 ZDC MC simulation (including Run3 ZDC TPs) does not exist, as there is still Run1 legacy ZDC Geometry and Digitization (+ legacy DB conditions) in CMSSW_14_1_X

[hjbossi]

Yes, that is also my understanding. Assuming that the geometry issues are fixed by @bsunanda, are there still other changes that we need to do for this?

[missirol]

• The necessary updates to include this in all the relevant sequences/workflows to re-emulate the L1T results, e.g. SimL1EmulatorRepack_Full_cff.

The example in [*] for CMSSW_14_1_0_pre5 shows that the L1T results before prescales are the same for hltData_ref.log (no L1T emulation) and hltData_L1uGT.log (L1REPACK:uGT), while all the ZDC L1T seeds accept zero events in hltData_L1Full.log (L1REPACK:Full).

Is the test in [*] with L1REPACK:Full expected not to work for ZDC seeds ? (cc: @eyigitba)

[*]

#!/bin/bash

# CMSSW_14_1_0_pre5

execmd="hltGetConfiguration /dev/CMSSW_14_0_0/HIon --no-prescale --no-output --max-events 100"
execmd+=" --paths HLTriggerFirstPath,HLTriggerFinalPath,HLTAnalyzerEndpath"

${execmd} \
  --customise HLTrigger/Configuration/CustomConfigs.customiseHLTforHIonRepackedRAWPrime \
  --input file:/eos/cms/store/user/cmsbuild/store/hidata/HIRun2023A/HIPhysicsRawPrime0/RAW/v1/000/375/491/00000/de963321-c0a0-49fb-b771-1a312a69db03.root \
  --globaltag 140X_dataRun3_HLT_v3 \
  --data \
  > hltData_ref.py && cmsRun hltData_ref.py &> hltData_ref.log

${execmd} \
  --customise HLTrigger/Configuration/CustomConfigs.customiseL1THLTforHIonRepackedRAWPrime \
  --input file:/eos/cms/store/user/cmsbuild/store/hidata/HIRun2023A/HIPhysicsRawPrime0/RAW/v1/000/375/491/00000/de963321-c0a0-49fb-b771-1a312a69db03.root \
  --globaltag 140X_dataRun3_HLT_v3 \
  --data \
  --eras Run3 --l1-emulator uGT \
  > hltData_L1uGT.py && cmsRun hltData_L1uGT.py &> hltData_L1uGT.log

${execmd} \
  --customise HLTrigger/Configuration/CustomConfigs.customiseL1THLTforHIonRepackedRAWPrime \
  --input file:/eos/cms/store/user/cmsbuild/store/hidata/HIRun2023A/HIPhysicsRawPrime0/RAW/v1/000/375/491/00000/de963321-c0a0-49fb-b771-1a312a69db03.root \
  --globaltag 140X_dataRun3_HLT_v3 \
  --data \
  --eras Run3 --l1-emulator Full \
  > hltData_L1Full.py && cmsRun hltData_L1Full.py &> hltData_L1Full.log

[eyigitba]

Hi @missirol , I would assume that L1REPACK:Full and L1REPACK:FullSimTP should re-emulate ZDC ETSum and trigger on this correctly. I think the reason for zero events passing is that something about the ZDC emulation workflow is missing in these sequences.

Let me ping @aloeliger @epalencia directly and hopefully they can have a look at this.

[hjbossi]

Hi All, Thanks for your help with this. Yes, it would be good to have a look and discuss as needed. We have some workforce that could help make the required changes as long as we have a good understanding of what is needed. Please let me know if it would be helpful to discuss offline.

[abdoulline]

I'm not sure whether
ZDC TPs emulation would work (as is) in CMSSW_14_0/14_1, even if uses only EM and HAD (existing since the beginning of ZDC deployment) channels.
E.g. for generating 2023 ZDC LUTs were used private txt conditions in a privately "hacked" CMSSW.

Actual ZDC DB conditions contain Run1 legacy constants and recently (14_0) ZDCDetId was changed.
@Michael-Krohn @JHiltbrand do you have an idea what'd happen if ZDC TPs emulation were run "as is"?

[Michael-Krohn]

I have not tried to run the ZDC TP emulation as is (with the Run1 legacy DB conditions and the emulation in 14_0), but I am not surprised that there are zero events passing the ZDC trigger.

In our current version of the condition HcalLutMetadata there are no ZDC channels. This condition is used to construct the output LUT for all channels, including the ZDC ones. Where specifically a calibration factor is grabbed from that file (https://github.com/cms-sw/cmssw/blob/master/CalibCalorimetry/HcalTPGAlgos/src/HcaluLUTTPGCoder.cc#L633) and multiplied to the output energy of each channel. With this value not set for the ZDC channels in the current conditions, I'm not sure what value is being multiplied here but if it defaults to 0 then the output energy for every EM and HAD channel in the emulation would be 0.

[missirol]

Afaiu, L1REPACK:FullSimTP includes the re-emulation of the TPs, but L1REPACK:Full does not (it starts from the unpacked TPs in data), and the latter is also not working (based on #43214 (comment)).

[missirol]

I think the reason for zero events passing is that something about the ZDC emulation workflow is missing in these sequences.

@eyigitba , I think that's indeed the case, see #45712. Note that even with #45712 the results of L1REPACK:uGT and L1REPACK:Full are not the same for L1T-ZDC seeds (and I did not investigate why, but someone should).

[aloeliger]

Okay, I finally have some time to work on this. @missirol, is this recipe: #43214 (comment) still accurate for recreating the problem?

[aloeliger]

• The necessary updates to include this in all the relevant sequences/workflows to re-emulate the L1T results, e.g. SimL1EmulatorRepack_Full_cff.

The example in [*] for CMSSW_14_1_0_pre5 shows that the L1T results before prescales are the same for hltData_ref.log (no L1T emulation) and hltData_L1uGT.log (L1REPACK:uGT), while all the ZDC L1T seeds accept zero events in hltData_L1Full.log (L1REPACK:Full).

Is the test in [*] with L1REPACK:Full expected not to work for ZDC seeds ? (cc: @eyigitba)

[*]

#!/bin/bash

# CMSSW_14_1_0_pre5

execmd="hltGetConfiguration /dev/CMSSW_14_0_0/HIon --no-prescale --no-output --max-events 100"
execmd+=" --paths HLTriggerFirstPath,HLTriggerFinalPath,HLTAnalyzerEndpath"

${execmd} \
  --customise HLTrigger/Configuration/CustomConfigs.customiseHLTforHIonRepackedRAWPrime \
  --input file:/eos/cms/store/user/cmsbuild/store/hidata/HIRun2023A/HIPhysicsRawPrime0/RAW/v1/000/375/491/00000/de963321-c0a0-49fb-b771-1a312a69db03.root \
  --globaltag 140X_dataRun3_HLT_v3 \
  --data \
  > hltData_ref.py && cmsRun hltData_ref.py &> hltData_ref.log

${execmd} \
  --customise HLTrigger/Configuration/CustomConfigs.customiseL1THLTforHIonRepackedRAWPrime \
  --input file:/eos/cms/store/user/cmsbuild/store/hidata/HIRun2023A/HIPhysicsRawPrime0/RAW/v1/000/375/491/00000/de963321-c0a0-49fb-b771-1a312a69db03.root \
  --globaltag 140X_dataRun3_HLT_v3 \
  --data \
  --eras Run3 --l1-emulator uGT \
  > hltData_L1uGT.py && cmsRun hltData_L1uGT.py &> hltData_L1uGT.log

${execmd} \
  --customise HLTrigger/Configuration/CustomConfigs.customiseL1THLTforHIonRepackedRAWPrime \
  --input file:/eos/cms/store/user/cmsbuild/store/hidata/HIRun2023A/HIPhysicsRawPrime0/RAW/v1/000/375/491/00000/de963321-c0a0-49fb-b771-1a312a69db03.root \
  --globaltag 140X_dataRun3_HLT_v3 \
  --data \
  --eras Run3 --l1-emulator Full \
  > hltData_L1Full.py && cmsRun hltData_L1Full.py &> hltData_L1Full.log

@missirol I am unable to recreate this particular check on lxplus for CMSSW_14_1_0_pre5. It fails with the error: Unable to find plugin 'HcalMahiConditionsESProducer@alpaka'. Is there antoher environment I should be running this under, or a different way to recreate the ZDC issue?

[makortel]

It fails with the error: Unable to find plugin 'HcalMahiConditionsESProducer@alpaka'

This error means that HeterogeneousCore.AlpakaCore.ProcessAcceleratorAlpaka_cfi is not loaded into the cms.Process (or the equivalent attachment of ProcessAcceleratorAlpaka object is not done). In an "offline configuration" that would be achieved either by loading Configuration.StandardSequences.Accelerators_cff, or Configuration.StandardSequences.Services_cff with alpaka modifier being enabled.

In the hltGetConfiguration case, should the necessary setup come via the HLT menu itself? I see the HLT_GRun_cff dump in CMSSW to load Configuration.StandardSequences.Accelerators_cff.

[missirol]

I think the issue is simpler. CMSSW_14_1_0_pre5 was cut before #44910 was merged, so it actually does not have that plugin. The recipe I posted is fragile in that it used /dev/CMSSW_14_0_0/HIon, and, when one does not specify a version number (e.g. /dev/CMSSW_14_0_0/HIon/V1), then /dev/CMSSW_14_0_0/HIon means "the latest version", so the output of the recipe is time-dependent.

I'll have a look, and provide an updated recipe.

In the hltGetConfiguration case, should the necessary setup come via the HLT menu itself?

I think so. The ConfDB db->python converter has some logic to add the proper loads to a given HLT configuration (see here if interested).

[mmusich]

In the hltGetConfiguration case, should the necessary setup come via the HLT menu itself?

yes.

I am unable to recreate this particular check on lxplus for CMSSW_14_1_0_pre5.

that happens because /dev/CMSSW_14_0_0/HIon is not the same menu it was back in July (see https://twiki.cern.ch/twiki/bin/view/CMS/ConfDB1400 for a bit of history). It have would have been better to give a particular version tag in the recipe that just the "HEAD" of the menu.
Technically the same recipe runs now in CMSSW_14_1_0.

[missirol]

@aloeliger

The updated recipe is in [*], tested with CMSSW_14_1_1. As mentioned in #43214 (comment), after #45712 the L1REPACK:Full case returns non-zero accepted events for ZDC-based L1T seeds, but the issue is that those trigger results differ from those of L1REPACK:uGT, and I do not know why that is.

[*]

#!/bin/bash

# CMSSW_14_1_1

execmd="hltGetConfiguration /dev/CMSSW_14_1_0/HIon/V26 --no-prescale --no-output --max-events 100"
execmd+=" --paths HLTriggerFirstPath,HLTriggerFinalPath,HLTAnalyzerEndpath"

${execmd} \
  --customise HLTrigger/Configuration/CustomConfigs.customiseHLTforHIonRepackedRAWPrime \
  --input file:/eos/cms/store/user/cmsbuild/store/hidata/HIRun2023A/HIPhysicsRawPrime0/RAW/v1/000/375/491/00000/de963321-c0a0-49fb-b771-1a312a69db03.root \
  --globaltag 140X_dataRun3_HLT_v3 \
  --data \
  > hltData_ref.py && cmsRun hltData_ref.py &> hltData_ref.log

${execmd} \
  --customise HLTrigger/Configuration/CustomConfigs.customiseL1THLTforHIonRepackedRAWPrime \
  --input file:/eos/cms/store/user/cmsbuild/store/hidata/HIRun2023A/HIPhysicsRawPrime0/RAW/v1/000/375/491/00000/de963321-c0a0-49fb-b771-1a312a69db03.root \
  --globaltag 140X_dataRun3_HLT_v3 \
  --data \
  --eras Run3 --l1-emulator uGT \
  > hltData_L1uGT.py && cmsRun hltData_L1uGT.py &> hltData_L1uGT.log

${execmd} \
  --customise HLTrigger/Configuration/CustomConfigs.customiseL1THLTforHIonRepackedRAWPrime \
  --input file:/eos/cms/store/user/cmsbuild/store/hidata/HIRun2023A/HIPhysicsRawPrime0/RAW/v1/000/375/491/00000/de963321-c0a0-49fb-b771-1a312a69db03.root \
  --globaltag 140X_dataRun3_HLT_v3 \
  --data \
  --eras Run3 --l1-emulator Full \
  > hltData_L1Full.py && cmsRun hltData_L1Full.py &> hltData_L1Full.log

[aloeliger]

@missirol Okay, I see the issue, and the surface level answer is both simple and unsatisfying. the uGT repack step is designed to only unpack uGT's inputs, rerun the trigger, then pack the inputs it just unpacked:

cmssw/Configuration/StandardSequences/python/SimL1EmulatorRepack_uGT_cff.py

Lines 24 to 50 in 4a85623

	simGtExtFakeStage2Digis.tcdsRecordLabel= cms.InputTag("unpackTcds","tcdsRecord")
	simGtStage2Digis.MuonInputTag = "unpackGtStage2:Muon"
	simGtStage2Digis.MuonShowerInputTag = "unpackGtStage2:MuonShower"
	simGtStage2Digis.EGammaInputTag = "unpackGtStage2:EGamma"
	simGtStage2Digis.TauInputTag = "unpackGtStage2:Tau"
	simGtStage2Digis.JetInputTag = "unpackGtStage2:Jet"
	simGtStage2Digis.EtSumInputTag = "unpackGtStage2:EtSum"
	simGtStage2Digis.EtSumZdcInputTag = "unpackGtStage2:EtSumZDC"
	simGtStage2Digis.CICADAScoreInputTag = "unpackGtStage2:CICADAScore"
	simGtStage2Digis.ExtInputTag = "unpackGtStage2" # as in default
	# Finally, pack the new L1T output back into RAW
	# pack simulated uGT
	from EventFilter.L1TRawToDigi.gtStage2Raw_cfi import gtStage2Raw as packGtStage2
	packGtStage2.MuonInputTag = "unpackGtStage2:Muon"
	packGtStage2.ShowerInputLabel = "unpackGtStage2:MuonShower"
	packGtStage2.EGammaInputTag = "unpackGtStage2:EGamma"
	packGtStage2.TauInputTag = "unpackGtStage2:Tau"
	packGtStage2.JetInputTag = "unpackGtStage2:Jet"
	packGtStage2.EtSumInputTag = "unpackGtStage2:EtSum"
	packGtStage2.EtSumZDCInputTag = "unpackGtStage2:EtSumZDC"
	packGtStage2.CICADAScoreInputTag = "unpackGtStage2:CICADAScore"
	packGtStage2.GtInputTag = "simGtStage2Digis" # as in default
	packGtStage2.ExtInputTag = "unpackGtStage2" # as in default

Full on the other hand unpacks the basic level of L1 emulation inputs (Calo L1 inputs, and a host of muon stuff, plus things like the ZDC) and then attempts to run the entire emulation chain up to uGT. There's some inaccuracies somewhere in how this emulation is done.

By nature, I am inclined to trust the thing with the fewest moving parts here, the uGT repack.

Digging a little deeper, and looking at the two logs, there are inaccuracies in SingleJet seeds, surprisingly no inaccuracies in single or double EG seeds, small inaccuracies in muon seeds (I saw 1 disagreement for L1_DoubleMuOpen_OS_BptxAND), as well as strange sum disagreements (centrality triggers, minimum bias stuff, and of course ZDC).

The muon and EG thing is telling. The thing the worst performers have in common is that they are all ultimately fed by unpacked HCAL trigger primitives, and the re-emulation starts from there.

cmssw/Configuration/StandardSequences/python/SimL1EmulatorRepack_Full_cff.py

Lines 127 to 129 in 4a85623

	# Calo Layer-1
	simCaloStage2Layer1Digis.ecalToken = 'unpackEcal:EcalTriggerPrimitives'
	simCaloStage2Layer1Digis.hcalToken = 'unpackHcal'

(Slightly curious to me here, is that this is unpackHcal, not unpackHcal:HcalTriggerPrimitives, but I don't imagine that it really matters)

cmssw/Configuration/StandardSequences/python/SimL1EmulatorRepack_Full_cff.py

Lines 131 to 132 in 4a85623

	# ZDC EtSums
	l1tZDCEtSums.hcalTPDigis = 'unpackHcal'

It is not out of the realm of possibility, that both calo triggers and ZDC emulation both have emulation inaccuracies, but the common factor here is the HCAL trigger primitive unpacking, so I suspect ultimately that what this is, is an HCAL unpacking bug.

The main portion of HCAL unpacker itself seems to have been unchanged for 4 years or so (https://github.com/cms-sw/cmssw/tree/master/EventFilter/HcalRawToDigi/plugins), so unless we've been suffering this for the last 4 years or so, we really need to get an HCAL expert to talk about what might have changed in the FED(s), or in the unpacking chain.

[aloeliger]

cmssw/Configuration/StandardSequences/python/SimL1EmulatorRepack_Full_cff.py

Lines 127 to 129 in 4a85623

	# Calo Layer-1
	simCaloStage2Layer1Digis.ecalToken = 'unpackEcal:EcalTriggerPrimitives'
	simCaloStage2Layer1Digis.hcalToken = 'unpackHcal'

(Slightly curious to me here, is that this is unpackHcal, not unpackHcal:HcalTriggerPrimitives, but I don't imagine that it really matters)

Actually, that does have an effect, which is odd to me. Changing it to unpackHcal:HcalTriggerPrimitives seems to make the agreement worse, and in seeds I wouldn't have expected like EG seeds. Changing unpackEcal:EcalTriggerPrimitives to unpackEcal just breaks everything.

[hjbossi]

Tagging HCAL trigger experts @JHiltbrand and @Michael-Krohn so that they can take a look.

[aloeliger]

After discussing with the DPG, we'll try looking at simulated and unpacked TPs, to see if that makes sense to us.