Add DBScan-based trigger algorithm

[TjarkMiener]

Please describe the use case that requires this feature.
For the proposed advanced SIPM-based LST camera (fully digitized) we explore new trigger algorithms based on digital sums and DBScan clustering. Preliminary tests show promising results. We would like this algorithm to be implemented in ctapipe, so we can run it as a software trigger on simtel with a very low initial trigger condition (almost "trigger-less") in order to evaluate the performance of the trigger system.

Describe the solution you'd like
Design an API for the SoftwareTrigger allowing different trigger algorithm.

Describe alternatives you've considered
We so far worked with "trigger-less" ctapipe hdf5 files, which is not handy.

[kosack]

Software triggering is more of a hack right now to deal with simulations, and it wasn't in the original scope of ctapipe. However, it would not be too hard to make it a selectable algorithm like we do with all other Components.

However, can you describe a bit more about what this algorithm should do? Is it a stereo array trigger algorithm, or a camera-level trigger? (the latter is not in the scope of ctapipe)

[burmist-git]

Dear Karl,
After discussing with Tjark and Matthieu, I’m providing more information below about the DBSCAN trigger, the DBSCAN algorithm for waveform (data cube and not images).
We are currently developing the SiPM camera for LST and considering a spatio-temporal trigger based on the DBSCAN algorithm. Preliminary studies have shown impressive performance using this DBSCAN-based trigger applied to the raw waveform digital sum. Since this development could serve any CTAO future instruments, we aim to use the official software for performance studies.
The DBSCAN trigger is capable of detecting very faint events (please refer to the attached GIF file sipm_cam_ev212404_faint.gif). In the GIF, the left panel shows the event development over time without NSB and with reduced electronic noise (by a factor of ~10). The central panel displays the same event with NSB and expected electronic noise. The right panel provides information about the gamma (its energy, core position, and other details). We apply the DBSCAN trigger on the central image(data cube) and display the triggered patch on the left "clean" image(data cube) in red.
As such it is indeed a monoscopic trigger, but we are also investigating its use as topological trigger, as the cluster identification can also serve as proxy to shower direction.
Initially, we intended to implement this trigger in sim_telarray, but it was advised by Konrad himself to implement this in a pipeline after sim_telarray.
We have all the code ready in C/C++ in case we decide to implement it in sim_telarray in the future. Their response suggested using the "triggerless" setup within sim_telarray with a ~20-30 p.e. cut and applying any algorithm for data volume reduction later. Following this, we learned that ctapipe has a software trigger, which can be utilized for this purpose.
The algorithm itself is straightforward and fast (please refer to the relevant article below). It can be used for waveform cleaning.
For waveform-level image cleaning, the algorithm can identify single clusters (please see the GIF file sipm_cam_ev7009805_long_in_time.gif, where the left panel highlights the cluster in red).
Implementation Idea:

We can implement the algorithm itself in the reducer (ctapipe/image/reducer.py) as a separate class or an inherited class.
We can temporarily call it in the software trigger.

Please note we are working with waveforms (data cube) and not with images described here :
PR dealing with this: #2401 (A time clustering algorithm for ima


ge cleaning #2401).

[kosack]

What is in ctapipe now is not really a trigger simulation, but rather just a way to simulate the effect of having a hardware trigger on the LSTs after we do a sub-selection of telescopes for a particular subarray. So what we do is not really a simulation of SWAT (the software array trigger), but just a correction to ensure an event that shouldn't have passed the hardware stereo trigger on the LSTs doesn't make it through for further analysis.

I'm not convinced that ctapipe is the right place to put such low-level trigger simulations, since it is intended to process data after triggering has occurred. However, I do agree that if the algorithm is used for waveform filtering to generate an image (which is in the DL0→DL1 transition), then it could naturally go into ctapipe. I think these are two separate use cases though.

One intermediate option for the first use case ( trigger-level simulation) might be to use ctapipe to write out R0 data (waveforms) in hdf5 format, and then simply filter that data with your own software, and use ctapipe to process the result further.

In any case, I think we should have a discussion about this in an upcoming ctapipe developer meeting (@maxnoe when is the next one? I guess in september), since I would like to keep the feature-creep of ctapipe down to a minimum, and to extend ctapipe to trigger-level simulation I think will need at least a CEP (ctapipe enhancement proposal) to be written.

[kosack]

We might also need a CEP written to support fancier methods that combine waveform to image generation with cleaning at the same time, such as what @burmist-git suggests above. That is not in the current design of ctapipe where image generation and cleaning are separated, and would require some refactoring to support properly. But there is certainly some interesting things that can be done there, and we have already discussed some similar algorithms in the past (and indeed I think EventDisplay already does something similar).

[GernotMaier]

For completeness: Eventdisplay uses a double pass trace integration and cleaning method which does:

• Pass 1: integrate waveforms with a wide integration window
• do image analysis
• Pass 2: integrate waveform with a optimal (very short) integration length placed at positions predicted by a fit to the time gradients along the length axis of the image

This method gets rid additionally of signal with times incompatible with a gamma-ray shower. Note that there is a tendency of making hadron showers more gamma-ray like (which is not good), but the advantages outweight this.

[TjarkMiener]

We might also need a CEP written to support fancier methods that combine waveform to image generation with cleaning at the same time, such as what @burmist-git suggests above. That is not in the current design of ctapipe where image generation and cleaning are separated, and would require some refactoring to support properly.

(Maybe too naive approach:) I have removed the label API change because I think we only need to change the returning shape of the select_pixels() call for the DVR components. Since it suppose to be working on waveforms the natural returning shape should be (n_samples, n_pixels) anyway. I think it is currently returning a single pixel mask is based on the TailCutsDataVolumeReducer, which is initially creating higher data levels to come up with a decision to keep the pixel or not. @kosack What do you think of adding a DBSCANDataVolumeReducer sub-component and changing the returning shape?

[kosack]

What do you think of adding a DBSCANDataVolumeReducer sub-component and changing the returning shape

That sounds reasonable, and isn't too much of an API change. Should ask the people working on DVR to be sure it's ok, but I don't see a major issue. That way you can sort of simulate the effect of a trigger by just applying DVR and filtering events that don't have any surviving pixels or something like that.

[maxnoe]

For completeness: Eventdisplay uses a double pass trace integration and cleaning method which does:

We do have that implemented in the TwoPassWindowSum image extractor:
https://ctapipe.readthedocs.io/en/latest/api/ctapipe.image.extractor.TwoPassWindowSum.html
though I think the implementation follows the MARS version of this, not the EventDisplay one

[maxnoe]

Since this development could serve any CTAO future instruments, we aim to use the official software for performance studies.

ctapipe is currently data levels R1 and upwards, ideally it would only be DL0 and up. The differences are small though (only pixels removed in DL0, otherwise the same as R1).

What you propose here is much more low level, and as you wrote really belongs into sim_telarray or something in-between sim_telarray and ctapipe.

Following this, we learned that ctapipe has a software trigger, which can be utilized for this purpose.

ctapipe does not have a software trigger at this level. It has an implementation of an array trigger that i used only to reject invalid telescope combinations that arise due to the simulation of hyper-arrays in the simulation. No timing, no looking at traces etc. Just looking at telescope ids in an already triggered hyper-array event and rejecting invalid combinations for the currently chosen realistic array.