High HMT rates with the latest L1NTuple recipe based on 130X

[elfontan]

In the last L1Ntuple procuctions based on CMSSW_13_0_X recipes, high HMT rates are extensively observed (in uGTTree/uGTEmuTree/L1MenuTools on data and uGTEmuTree/L1MenuTools in MC), never observed on all the previous productions based on CMSSW_12_6_X.

Note that the OSW IB was rebased twice recently:

• CMSSW_13_0_0_pre2
• CMSSW_13_0_0_pre4

Previously we were working in:

• CMSSW_12_6_0_pre1

Some numbers are reported as reference below (extracted from run 362429 with levelling at PU 70):

126X:
L1_SingleMuShower_Nominal: 3250.51 Hz (718 countings)
L1_SingleMuShower_Tight: 1507.55 Hz (333 countings)
130X:
L1_SingleMuShower_Nominal: 21265.6 Hz (4680 countings)
L1_SingleMuShower_Tight: 8088.2 Hz (1780 countings)

The expected rate from data during the 2022 data-taking period is very close to what we observe in 126X L1Ntuples.
The weird thing is that we observe the same high rates both in unpacked and emu rates.

Comparing L1MenuTools and uGT emulator on a subset of events after the updates of the UGT emulator, we have:

L1_SingleMuShower_Nominal -> uGTEmuTree 233 = L1MenuTools 233 (uGTTree: 231)
L1_SingleMuShower_Tight -> uGTEmuTree 108 = L1MenuTools 108 (uGTTree: 107)

So that L1MenuTools and uGT provide coherent results and unpacked rates are pretty close to emulated rates.

Additional Notes:

• List of CMSSW changes with milestone CMSSW_12_6_X
• List of CMSSW changes with milestone CMSSW_13_0_X

Recipe used to produce L1Ntuples in the two cases (5k events):

cmsDriver.py l1Ntuple -s RAW2DIGI --python_filename=data_def.py -n -1 --no_output --era=Run3 --data --conditions=124X_dataRun3_Prompt_v4 --customise=L1Trigger/Configuration/customiseSettings.L1TSettingsToCaloParams_2022_v0_6 --customise=L1Trigger/Configuration/customiseReEmul.L1TReEmulFromRAW --customise=L1Trigger/L1TNtuples/customiseL1Ntuple.L1NtupleRAWEMU --filein=/store/data/Run2022G/EphemeralZeroBias0/RAW/v1/000/362/439/00000/3d071811-fc6f-4375-8918-3839c1f5ea8c.root --fileout=L1Ntuple.root

Result (useful reference for debugging purposes):

• 130X: 5 showers (4 nominal, 1 tight)
• 126X: 0 showers

A slightly adapted recipe (without customisations related to the OSW IB) was used to track down the release where the higher countings started to occur:

cmsDriver.py l1Ntuple -s RAW2DIGI --python_filename=data_def.py -n -1 --no_output --era=Run3 --data --conditions=124X_dataRun3_Prompt_v4 --customise=L1Trigger/Configuration/customiseReEmul.L1TReEmulFromRAW --customise=L1Trigger/L1TNtuples/customiseL1Ntuple.L1NtupleRAWEMU --filein=/store/data/Run2022G/EphemeralZeroBias0/RAW/v1/000/362/439/00000/3d071811-fc6f-4375-8918-3839c1f5ea8c.root --fileout=L1Ntuple.root

The following releases have been tested:

• CMSSW_13_0_0_pre1
• CMSSW_12_6_4
• CMSSW_12_6_2
• CMSSW_12_6_0
• CMSSW_12_6_0_pre1
• CMSSW_12_6_0_pre2

The change happens between CMSSW_12_6_0_pre1 and CMSSW_12_6_0_pre2.
The list of changed files between the two releases is reported here. Looking at the list of PRs reported also above with milestone CMSSW_12_6_X, the only one that might affect the HMT behaviour looks PR#39318.

Follow up needed between the L1 OSW team (@aloeliger @epalencia) and @Sam-Harper and Martin/Pedro.

[aloeliger]

@elfontan I'll take a look at this as soon as I can, but that may be after the trigger review.

[aloeliger]

@elfontan I'm having trouble recreating this, even after pulling the relevant 12_6_0 and 13_0_0 integration branches.

I get this error:

----- Begin Fatal Exception 02-Apr-2023 05:46:16 CDT-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named data_def.py
Exception Message:
 unknown python problem occurred.
RuntimeError: An exception of category 'FileInPathError' occurred.
Exception Message:
edm::FileInPath unable to find file L1Trigger/L1TCalorimeter/data/egRecalibratedLUTv1_2022_v0_2.txt anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /afs/hep.wisc.edu/cms/aloeliger/L1Offline/HMTRates/CMSSW_12_6_0_pre1/poison:/afs/hep.wisc.edu/cms/aloeliger/L1Offline/HMTRates/CMSSW_12_6_0_pre1/src
:/afs/hep.wisc.edu/cms/aloeliger/L1Offline/HMTRates/CMSSW_12_6_0_pre1/external/slc7_amd64_gcc10/data:/cvmfs/cms.cern.ch/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_6_0_pre1/src:/cvm
fs/cms.cern.ch/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_6_0_pre1/external/slc7_amd64_gcc10/data
Current directory is: /afs/hep.wisc.edu/cms/aloeliger/L1Offline/HMTRates/CMSSW_12_6_0_pre1/src

The series of steps I performed are (for 12_6_0):

cmsrel CMSSW_12_6_0_pre1
cd CMSSW_12_6_0_pre1/src/
cmsenv
git cms-init
git cms-rebase-topic -u cms-l1t-offline:l1t-integration-CMSSW_12_6_0_pre1
scram b -j 8
<your cmsDriver command>

Could you repost the exact setup for your CMSSW version?

[elfontan]

Hi @aloeliger, at the first stage I was just using the two OSW recipes for the corresponding IBs as they were here, for example you can compare:

CMSSW_13_0_0_pre4

cmsrel CMSSW_13_0_0_pre4
cd CMSSW_13_0_0_pre4/src
cmsenv
git cms-init
git remote add cms-l1t-offline [email protected]:cms-l1t-offline/cmssw.git
git fetch cms-l1t-offline l1t-integration-CMSSW_13_0_0_pre4
git cms-merge-topic -u cms-l1t-offline:l1t-integration-v147
git clone https://github.com/cms-l1t-offline/L1Trigger-L1TCalorimeter.git L1Trigger/L1TCalorimeter/data
git cms-checkdeps -A -a
scram b -j 8

vs CMSSW_12_6_0_pre1

cmsrel CMSSW_12_6_0_pre1
cd CMSSW_12_6_0_pre1/src
cmsenv
git cms-init
git remote add cms-l1t-offline [email protected]:cms-l1t-offline/cmssw.git
git fetch cms-l1t-offline l1t-integration-CMSSW_12_6_0_pre1
git cms-merge-topic -u cms-l1t-offline:l1t-integration-v141
git clone https://github.com/cms-l1t-offline/L1Trigger-L1TCalorimeter.git L1Trigger/L1TCalorimeter/data
git cms-checkdeps -A -a
scram b -j 8

I think that you are just missing the L1Trigger/L1TCalorimeter/data repo.
Btw, side comment: I think that it might be helpful to keep at least the previous recipe (or previous two) in the OSW Twiki, even if hidden. Or at least in our gDocs.

[elfontan]

Dear @aloeliger @epalencia all,
I updated the description of the issue above with some further information after the debugging to track down the origin of this issue. Please have a look when you have time.
I am going to send a email to include various people interested in this.
(I am not able to tag most of them here for some reason...)

Cheers,
--Elisa

[elfontan]

Deaar @aloeliger @epalencia,
note that the high HMT rates issue was understood and fixed in the past months.
You can find some description of the issue in S5-6 of this presentation given at the Menu meeting on June 2.

To summarize:

1. The issue was tracked down using the full emulation in the L1Ntuples workflow: it started to appear in CMSSW_12_6_0_pre2. It was due to a bunch of updates at CSC level (see list of commits)
2. The important change that affected the shower counting downstream is that the updated CSC emulator outputs a vector of HMT decisions, one for each of 8 BXs. The EMTF emulator was integrating the full 8BXs to count the
   number of HMT instead of considering the single BX.

Find here the reference to the main emulator updates:

• Update if the EMTF muon shower unpacker: EMTF muon shower unpacker update cms-sw/cmssw#41993 + Prevent producing EMTFTrack if only muon shower was unpacked cms-sw/cmssw#42172
• Correct BX assignment: Change muon shower BX assignment in EMTF cms-sw/cmssw#42176

Thanks a lot Efe for the work and help in debugging and validating!
I think that the issue can be closed.

Cheers,
--Elisa