Reader cannot read runs with a single gain

[sizun]

Related to #15. Current NectarCAMEventSource cannot even read runs like run 3889 with a single gain (where gain selection is enabled for all trigger bits).

[tibaldo]

Thank you for reporting this. It should be easy to fix. I can look into it next week if there isn't anybody beating me to it ;)

[tibaldo]

This is on hold until the pull request addressing issue #32 is merged

[tibaldo]

Hi @sizun, could you please be more specific about the problem you are experiencing? I tried reading run 3889 with the current version of the reader and could do it successfully.

from ctapipe.io import EventSource, EventSeeker
path = 'NectarCAM.Run3889.0000.fits.fz'
reader=EventSource(input_url=path)
seeker = EventSeeker(reader)
evt = seeker.get_event_index(25)
import numpy as np
np.shape(evt.r1.tel[0].waveform)
import matplotlib.pyplot as plt
plt.plot(evt.r1.tel[0].waveform[1500])

The waveform shape is (1855,60), as expected after gain selection, and the plotted waveform looks reasonable.

The only issue I see is that the waveform stored in the r1 container is not pedestal-subtracted nor gain/flatfield corrected. This is because, in principle, the camera server will eventually apply all these calibrations along with gain selection. Thus, the logic in the code implementation was that if gain selection has already happened then the data are already calibrated as well. I'm not sure if this needs to be changed.

However, if there is a more fundamental problem in reading the files please let me know and I'll look into it. I suspect there is no fundamental difference for runs where only some events have gain-selection according to their trigger type because the r0 and r1 containers are filled on an event-by-event basis.

[sizun]

Hello @tibaldo
So far I have only ever used the r0 container.
For this run, the r0 container waveform is None.

from ctapipe.io import EventSource, EventSeeker
path = 'NectarCAM.Run3889.0000.fits.fz'
reader=EventSource(input_url=path)
seeker = EventSeeker(reader)
evt = seeker.get_event_index(25)
print(evt.r0.tel[0].waveform is None)

[tibaldo]

Hi @sizun, yes, that's what I explained. The logic in the code is that if gain selection has already happened than the entire r0->r1 calibration has already happened and the content of the file is dumped into the r1 container. The r0 container is not filled because the code assumes there are no r0 data if the gain selection has been applied.
You can just use the r1 container.

[sizun]

I still do not understand the contents of evt.r1.tel[0].waveform. The 60 samples plotted above are not integers, why? Has any processing been applied to the data stored in the ZFITS file?

[tibaldo]

Well spotted! I checked calibration.py and realized that by the way the code is written the default flatfield coefficients are applied in this case. While we think of a possible improvement in the code conception you can read the raw waveforms by disabling all the r0->r1 corrections.

In [1]: from ctapipe.io import EventSource, EventSeeker

In [2]: from traitlets.config import Config

In [3]: path = 'NectarCAM.Run3889.0000.fits.fz'

In [4]: config = Config(dict(
   ...:     NectarCAMEventSource=dict(
   ...:         NectarCAMR0Corrections=dict(
   ...:         calibration_path = None,
   ...:         apply_flatfield = False,
   ...:         select_gain = False,
   ...: ))))

In [5]: reader=EventSource(input_url=path,config = config)

In [6]: seeker = EventSeeker(reader)

In [7]: evt = seeker.get_event_index(25)
Seeking event by iterating through events.. (potentially long process)

In [8]: evt.r1.tel[0].waveform[1500]
Out[8]: 
array([253, 255, 258, 252, 254, 253, 247, 249, 248, 251, 250, 259, 255,
       255, 257, 254, 249, 250, 254, 252, 247, 250, 253, 252, 250, 253,
       251, 252, 252, 254, 254, 255, 252, 257, 258, 253, 254, 253, 250,
       250, 253, 254, 254, 256, 254, 254, 253, 254, 253, 258, 258, 253,
       254, 251, 249, 253, 251, 252, 252, 253], dtype=uint16)

Please let me know if this works for you.

[jlenain]

Hello @tibaldo , @sizun ,
This is also seen to affect the Data Quality Monitoring for such runs on the nectarchain. Have you had the time to devise a possible way forward on this?
Thanks a lot!

[tibaldo]

Hello @jlenain
As explained in my answer above the issue described in the title does not exist. The files are read as they are intended to be, that is, if gain selection is applied the code assumes that r1 calibration already happened.
A quick fix for the DQM is probably to disable r0->r1 calibration and read the r1 container if the r0 container is not available.
Please, close this issue.
If you want to change the way files with gain-selection are read, please open another issue and specify there the corresponding requirements.

[jlenain]

Hi @tibaldo ,
Thanks a lot for the clarification. Indeed, the R0->R1 calibration should then be disabled in the DQM.