New implementation of the nectarchain module (#82)

* clean commit history :
-update SPE fit parameters initialization
following updates of the ctapipe extractors
-improve the limits of generated charge histogram plots

* test script

* add method to sort WaveformsContainer
and ChargesContainer following events_id

* -waveformContainer contructor from event list

* user script for test ggroller

* change values of pp and n in fit parameters file
 by median

* improve test

* repare test

* reduce pvalue digit in label

* -add time counter to SPE fit

* add possibility to freez parameters into minuit

* begin : redesign of the SPE fit class to speed up

* new skeleton for the calibration pipeline :
-container module for the data structure (to be extend)
-makers module to make the computation of calibration quantities
 on the data structure

* mothers class for the maker module

* mother class for the gain makers

* unchanged modeuls for the new structure

* FlatFieldSPEMaker implementation

* photostatistic maker (not re-implemented yet)

* implementation of the FFSPE fit for single run
with n and p' fixed

* implementation of FF SPE gain maker
at nominal voltage
+ update of user script

* bugfix
user script updated : FF SPE at VVH voltage (ie for single run)

* update for nominal voltage data FF SPE fit from HHV
pre-computed one

* -bugfix
update for nominal FF SPE fit

* put I/O method in mother class

* add low gain in contructor (only high gain before)

* typo bugfiox

* move gain utils in one single folder

* bugfix nominal voltage fit
the fitting process continues although the initial
 parameters computation failed

* Photostatistic implementation done

* cleaning : import, type, etc

* remove old SPE fit implementation

* makers : core module enhancement
 (docstrings, unit test)

* cleaning

* moving container and makers in
 top of calibration directory

* folder restructuring

* cleaning after the restructuration

* unit test for chargecontainer

* test extractor utlis

* unit test CalibrationMaker and GainMakers

* unit test flatfieldSPEMakers

* unit test implementation follow up

* mv wavefomrs in makers

* code into WavefomContainers split into 2 parts : one for the data format,
the other for the computation

* move charge container into makers module

* creation of mother class to charge and waveform container

* follow up : factorisation waveforms and chargs Makers

* display module

* finalisation of the refactorisation of the waveformsMaker
chargesMaker implementation

* ChargesContainers I/O
cleaning
unit test for WaveformsMaker and ChargesMaker

* update calibration class following COntainers changes

* bugfix according to unit tests

* docstrings container + unit test

* make container module visible

* small bugfix + fix unit test for makers

* docstrings makers module

* follow pre-commit hook

* set Dirac import into methods which need it

* first step to reimplement following ctapipe
 Tool and Components framework

* -removed the I/O method for containers
    -> now we will use ctapipe HDF5TableWriter and Reader
-creation of recursive containers

* -adapt charges extraction makers and waveform extraction
makers with the ctapipe.Tool framework

* -creation of the component module, which inherits from ctapipe.component
-implementation of the NectarCAMComponent, the base class for nectrachain components,
- implementation of the ChargesComponent and WaveformsComponent

* -preliminary tuto script to extract charges
or waveforms using Tool and Component classes

* renaming code name for gain
computation (makers->Component)

* move SPE code in component module

* move SPE algo in utlis

* renaming spe module

* changement yaml parameters spe location

* mv spe algo in component main folder

* move utils outside of the sub-modules

* charge and waveforms extraction with ctapip Tool

* container for charges and waveforms

* fast eventsource reader (30% faster)

* gain container to store interesting values
 computed from gain calibration

* for spe algorithm Component

* calibration module

* calibration module : SPE fit implementation

* add useful method

* utils for nectarchain

* script to load waveforms and extract charge updated

* renaming code name for gain
computation (makers->Component)

* move SPE code in component module

* move SPE algo in utlis

* renaming spe module

* changement yaml parameters spe location

* mv spe algo in component main folder

* move utils outside of the sub-modules

* charge and waveforms extraction with ctapip Tool

* container for charges and waveforms

* fast eventsource reader (30% faster)

* gain container to store interesting values
 computed from gain calibration

* for spe algorithm Component

* calibration module

* calibration module : SPE fit implementation

* add useful method

* utils for nectarchain

* script to load waveforms and extract charge updated

* cleaning

* container namesape + other

* waveforms from hdf5 method

* SPE fit for flatfirld runs implementation

* start of reimplementatuion of phtotstat method

* follow up photostat implementation

* clean way to manage the work done on reduced
part of the raw data (with max_events parameter)

* forgotten in last commit..

* photostatistic re-implemented

* same

* typo changes

* circular import bugfix

* imprve log

* bugfix output filename

* bugfix I/O + maxevents read written now in output file
+ data management with max_events parameter

* some bugfix

* cleaning and improvement of the data management

* updates of scripts

* remove old unit test

* update runs of interest

* bugfix :
-find SPE data
-containers data with maxevents

* load Containers : now if it is splited in slices,
will return a generator to only
 load the slices one by one during execution
 -> reduce the RAM needed

* bugfix computation of convergence rate

* -add classes to perform SPE fit directly
at nominal voltage (config files)
-change SPE fit plotting matplotlib -> pyqtgraph
 to speed up the plots generation

* -ignore png files
-bugfix : check valididty of container before to write when
finishing component

* update scripts

* tuto for :
-creation of your own component and tool
-SPE tools use
-exctraction of waveforms and chgares with tools and components.

* -fix issue to access component defined outside
of the nectarchain module
-typo fix
-overwrite argument of tool better defined

* bugfix : issue mentioned by @jlenain in #82.
Now the read of a container will always return a generator,
which can be ciomposed by only one item.

* update with pre-commit hooks

* ignore notebooks

* rm notebook

---------

Co-authored-by: guillaume.grolleron <[email protected]>

.gitignore

@@ -92,6 +92,7 @@ src/nectarchain/user_scripts/**/test
 **.log
 **.pdf
 **.csv
+**.png
 
 #VScode
 .vscode/

notebooks/tool_implementation/tuto_SPE.py

@@ -0,0 +1,75 @@
+# ---
+# jupyter:
+#   jupytext:
+#     text_representation:
+#       extension: .py
+#       format_name: percent
+#       format_version: '1.3'
+#       jupytext_version: 1.14.6
+#   kernelspec:
+#     display_name: nectarchain
+#     language: python
+#     name: python3
+# ---
+
+# %%
+import logging
+import os
+import pathlib
+
+logging.basicConfig(
+    format="%(asctime)s %(name)s %(levelname)s %(message)s", level=logging.INFO
+)
+log = logging.getLogger(__name__)
+log.handlers = logging.getLogger("__main__").handlers
+
+from nectarchain.makers.calibration import (
+    FlatFieldSPEHHVStdNectarCAMCalibrationTool,
+    FlatFieldSPENominalStdNectarCAMCalibrationTool,
+)
+
+# %%
+run_number = 3936
+
+# %%
+
+# !ls -lh $NECTARCAMDATA/runs/*
+
+# %%
+tool = FlatFieldSPENominalStdNectarCAMCalibrationTool(
+    progress_bar=True,
+    method="LocalPeakWindowSum",
+    extractor_kwargs={"window_width": 12, "window_shift": 4},
+    multiproc=True,
+    nproc=10,
+    run_number=run_number,
+    max_events=10000,
+    log_level=20,
+    reload_events=True,
+    # events_per_slice = 200,
+    asked_pixels_id=[52, 48],
+    output_path=pathlib.Path(os.environ.get("NECTARCAMDATA", "/tmp"))
+    / "tutorials/"
+    / f"SPEfit_{run_number}.h5",
+)
+
+# %%
+tool
+
+# %%
+tool.initialize()
+
+# %%
+tool.setup()
+
+# %%
+tool.start()
+
+# %%
+output = tool.finish(return_output_component=True, display=True, figpath=os.getcwd())
+output
+
+# %%
+output[0].resolution
+
+# %%