Adding a function to create a log file and save cloud correction para… #260

nzywucka · 2024-09-13T09:44:51Z

…meters in the file

codecov · 2024-09-13T09:56:14Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.18%. Comparing base (f9dca14) to head (d213c9b).
Report is 42 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #260      +/-   ##
==========================================
- Coverage   77.23%   77.18%   -0.06%     
==========================================
  Files          21       22       +1     
  Lines        2614     2621       +7     
==========================================
+ Hits         2019     2023       +4     
- Misses        595      598       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

jsitarek · 2024-09-13T12:50:22Z

I think you did it well too complicated. You already have one instance of "logging": logger that can be used, you just need to print out in the general log cloud_correction parameters from the config file (and also you can print out MCP version if you want, but this I think we should change such that it is done for all the scripts)

…s improved.

jsitarek · 2024-09-17T08:33:31Z

magicctapipe/scripts/lst1_magic/lst_m1_m2_cloud_correction.py

@@ -248,7 +252,7 @@ def process_telescope_data(input_file, config, tel_id, camgeom, focal_eff):
        if assigned_tel_ids["LST-1"] == tel_id:
            conc_params = concentration_parameters(
                clean_camgeom, clean_image, hillas_params
-            )  # "For LST-1 we compute concentration from the cleaned image and for MAGIC from the full image to reproduce the current behaviour in the standard code
+            )  # "For LST-1 we compute concentration from the cleaned image and for MAGIC from the full image to reproduce the current behaviour in the standard code/home/natti/CTA/lodz_cluster/lstchain_10_11_test/scripts/new/outfiles/dl1_coincidence_stereo_0.75-0.85_03955


this looks like a copy/paste error

jsitarek · 2024-09-17T08:43:24Z

magicctapipe/scripts/lst1_magic/lst_m1_m2_cloud_correction.py

-    subarray_info.to_hdf(args.output_file)
+    logging.info(f"Correction parameters: {correction_params}")
+    logging.info(f"ctapipe version: {ctapipe.__version__}")
+    logging.info(f"magicctapipe version: {magicctapipe.__version__}")


why do you use a separate log? you have "logger" defined before and used just below this line. you can use it also to write the correction parameters and version of software

…ogger improvements, and units changes.

jsitarek · 2024-10-02T02:02:52Z

Hi @nzywucka, the latest changes seem to went into another file.

jsitarek · 2024-10-07T06:35:03Z

Hi @nzywucka
With the fix of Alessio I rerun the tests and now they are successful, but there are still multiple unresolved comments (and somewhat on the way you changed the name of the script, so you have some of the changes implemented in a second file).

extracted_lidar_reports_test.csv: file with the cloud parameters from LIDAR lst1_magic_cloud_correction.py: interpolation function has been added to the code

jsitarek · 2024-11-08T06:18:51Z

magicctapipe/resources/config.yaml

@@ -102,7 +102,7 @@ event_coincidence:


 stereo_reco:
-    quality_cuts: "(intensity > 50) & (width > 0)"
+    quality_cuts: "(intensity > 50) & (intensity > 80 or tel_id > 1) & (width > 0)" #"(intensity > 50) & (width > 0)"


this cut was used for tests. it should not go into the default input card

jsitarek · 2024-11-08T06:20:59Z

magicctapipe/scripts/lst1_magic/extracted_lidar_reports_test.csv

@@ -0,0 +1,7 @@
+timestamp,base_height,top_height,transmission,lidar_zenith


better to put this file in resources directory (where the input cards are)

also, is this file still used for something? the yaml file has a different format

jsitarek · 2024-11-08T06:21:52Z

magicctapipe/resources/config.yaml

@@ -280,8 +280,8 @@ dl2_to_dl3:

 cloud_correction:
    specify_cloud: yes


it is not anymore "specify_cloud" option and it is not used anywhere in the code. I think you can drop it for the moment

magicctapipe/scripts/lst1_magic/lst1_magic_cloud_correction.py

lst1_magic_cloud_correction.py: interpolation implementation has been added to the code.

lidar_reports_clouds_ST.03.16.yaml: configuration file with LIDAR reports of ST.03.16 period added lst1_magic_cloud_correction.py: additional cleaning procedure and improvements to the interpolation function added

jsitarek · 2024-12-11T07:26:38Z

magicctapipe/resources/config.yaml

    number_of_layers: 10

+    additional_lst_cleaning:
+        picture_thresh: 10


the purpose of redoing the cleaning is to raise the thresholds by some factor, and the same factor should be applied to LST and MAGIC. So, either the factor is determined automatically in the script (but this can be messy because then one needs to process the simulations for each run even if changes are small, or you specify in the input card only a single multiplication factor and this factor is applied to picture_thresh, boundary_thresh of both MAGIC and LST, and rest of the settings should be taken from default config or from the file itself (at least for MAGIC, because for LST this can be more tricky)

jsitarek · 2024-12-11T08:57:17Z

magicctapipe/resources/lidar_reports_clouds_ST.03.16.yaml

@@ -0,0 +1,533 @@
+units:
+  timestamp: 'iso'


this file is for a very specific dataset, therefore there is no point of posting it into the general repository

lst1_magic_cloud_correction.py: update on the additional cleaning method

jsitarek · 2024-12-19T09:21:35Z

magicctapipe/scripts/lst1_magic/lst1_magic_cloud_correction.py

+        A tuple containing interpolated or nearest values for:
+        - base_height (float): The base height of the cloud layer in meters.
+        - top_height (float): The top height of the cloud layer in meters.
+        - vertical_transmission (float): Transmission factor of the cloud layer adjusted for vertical angle.


... for zenith angle

jsitarek · 2024-12-19T14:01:27Z

magicctapipe/scripts/lst1_magic/lst1_magic_cloud_correction.py

+    config_clean_LST = {
+        key: (
+            value * cmf
+            if isinstance(value, (int, float)) and not isinstance(value, bool)


this way you will upscale all the parameters, including e.g. min_number_picture_neighbours that should not be touched. It is better and simpler to directly upscale only picture_thresh and boundary_thresh

jsitarek · 2024-12-19T14:04:10Z

magicctapipe/scripts/lst1_magic/lst1_magic_cloud_correction.py

+
+    # AC MAGIC
+    # Ignore runtime warnings appeared during the image cleaning
+    warnings.simplefilter("ignore", category=RuntimeWarning)


even while this is copied from the magic_calib_do_dl1 I think it would be better to try to run without this and see if there are no issues. Because it might also hide some other warnings related to different parts of the code

jsitarek · 2024-12-19T14:05:41Z

magicctapipe/scripts/lst1_magic/lst1_magic_cloud_correction.py

+    config_clean_magic = {
+        key: (
+            False
+            if key == "find_hotpixels"


save as above, you should only scale picture_thresh and boundary_thresh, and do not touch other parameters

jsitarek · 2024-12-19T14:14:53Z

magicctapipe/scripts/lst1_magic/lst1_magic_cloud_correction.py

+        peak_time = dl1_images["peak_time"][index_img]
+        image /= trans_pixels
+
+        clean_image = image[clean_mask]


you use the old mask first no matter if you do the additional cleaning a moment later or not. I think it is faster to move those 3 lines as "else" after the next condition

jsitarek · 2024-12-19T14:19:58Z

magicctapipe/scripts/lst1_magic/lst1_magic_cloud_correction.py

+                    unsuitable_mask=unsuitable_mask,
+                )
+
+                # n_pixels = np.count_nonzero(clean_mask)


some leftover commented out lines here

jsitarek · 2024-12-19T14:21:17Z

magicctapipe/scripts/lst1_magic/lst1_magic_cloud_correction.py

+        prefixed_conc_params = {
+            f"concentration_{key}": value for key, value in conc_params.items()
+        }
+        event_params.update(prefixed_conc_params)


prefixed_conc_params could not be included directly in event_params a few lines above?

magicctapipe/scripts/lst1_magic/lst1_magic_cloud_correction.py

jsitarek · 2024-12-19T14:28:23Z

magicctapipe/scripts/lst1_magic/lst1_magic_cloud_correction.py

+
+    cloud_params = lidar_cloud_interpolation(
+        mean_subrun_timestamp, max_gap_lidar_shots, lidar_report_file
+    )


this function can return None if there are no proper LIDAR raports to interpolate from that would crash the code below. A better way to handle error management would be to introduce a new error here:
https://github.com/cta-observatory/magic-cta-pipe/blob/73e4a3bf47ace47deb11d1d74a4caf0c37954841/magicctapipe/utils/error_codes.py
and throw it like here:

magic-cta-pipe/magicctapipe/scripts/lst1_magic/lst1_magic_stereo_reco.py

Line 234 in 73e4a3b

sys.exit(NO_EVENTS_WITHIN_MAXIMUM_DISTANCE)

jsitarek · 2024-12-19T14:29:54Z

magicctapipe/scripts/lst1_magic/lst1_magic_cloud_correction.py

+    correction_params = config.get("cloud_correction", {})
+    max_gap_lidar_shots = u.Quantity(
+        correction_params.get("max_gap_lidar_shots")
+    )  # set it to 900 seconds


the comment is misleading - the code will set it to whatever is in the input card

jsitarek · 2024-12-19T14:34:54Z

magicctapipe/scripts/lst1_magic/lst1_magic_cloud_correction.py

+            try:
+                dl1_images = read_table(args.input_file, image_node_path)
+            except tables.NoSuchNodeError:
+                logger.info(f"\nNo image for telescope with ID {tel_id}. Skipping.")


this skips the telescopes without stored images and it is very dangerous because images are stored separately in MAGIC and in LST files, so e.g. if you have LST file without images, but MAGIC with images the script will make MAGIC-only analysis that is likely not what the user wanted.
So I would turn this into a fatal error

jsitarek · 2024-12-19T14:38:56Z

magicctapipe/scripts/lst1_magic/lst1_magic_cloud_correction.py

+                logger.info(f"\nNo image for telescope with ID {tel_id}. Skipping.")
+                dl1_images = None
+
+            df = process_telescope_data(


you pass the config to the function, so the parameters like tel_ids, nlayers, cmf should be recovered from the config inside this function instead of being passed additionally.

jsitarek · 2024-12-19T14:43:23Z

magicctapipe/scripts/lst1_magic/lst1_magic_cloud_correction.py

+
+    subarray_info.to_hdf(output_file)
+
+    correction_params = config.get("cloud_correction", {})


you already extracted it in L583, why do you do it again?

jsitarek · 2024-12-19T14:44:21Z

magicctapipe/scripts/lst1_magic/lst1_magic_cloud_correction.py

+    cloud_params = lidar_cloud_interpolation(
+        mean_subrun_timestamp, max_gap_lidar_shots, lidar_report_file
+    )
+


also I think it would be useful to print the cloud_params in the logger

Adding a function to create a log file and save cloud correction para…

1b55522

…meters in the file

Saving parameters to the .log file simplified and saving the .h5 file…

ba5ed61

…s improved.

jsitarek reviewed Sep 17, 2024

View reviewed changes

Provided changes in check of the available images in the .h5 files, l…

e0ce62d

…ogger improvements, and units changes.

nzywucka added 2 commits October 29, 2024 21:41

config.yaml: the cloud_correction section has been changed

bd16f73

extracted_lidar_reports_test.csv: file with the cloud parameters from LIDAR lst1_magic_cloud_correction.py: interpolation function has been added to the code

Rename file to correct name

edb06c7

jsitarek reviewed Nov 8, 2024

View reviewed changes

magicctapipe/scripts/lst1_magic/lst1_magic_cloud_correction.py Show resolved Hide resolved

nzywucka added 3 commits December 5, 2024 12:33

Removed filename from the repository

7d24ab2

config.yaml: stereo reco quality cut has been changed to the orginal.

8047de9

lst1_magic_cloud_correction.py: interpolation implementation has been added to the code.

config.yaml: additional cleaning LST and MAGIC parameters added

11041ed

lidar_reports_clouds_ST.03.16.yaml: configuration file with LIDAR reports of ST.03.16 period added lst1_magic_cloud_correction.py: additional cleaning procedure and improvements to the interpolation function added

jsitarek reviewed Dec 11, 2024

View reviewed changes

config.yaml: modification of the cloud_correction section

d213c9b

lst1_magic_cloud_correction.py: update on the additional cleaning method

jsitarek reviewed Dec 19, 2024

View reviewed changes

magicctapipe/scripts/lst1_magic/lst1_magic_cloud_correction.py Show resolved Hide resolved

jsitarek reviewed Dec 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding a function to create a log file and save cloud correction para… #260

Adding a function to create a log file and save cloud correction para… #260

nzywucka commented Sep 13, 2024

codecov bot commented Sep 13, 2024 •

edited

Loading

jsitarek commented Sep 13, 2024

jsitarek Sep 17, 2024

jsitarek Sep 17, 2024

jsitarek commented Oct 2, 2024

jsitarek commented Oct 7, 2024

jsitarek Nov 8, 2024

jsitarek Nov 8, 2024

jsitarek Dec 19, 2024

jsitarek Nov 8, 2024

jsitarek Dec 11, 2024

jsitarek Dec 11, 2024

jsitarek Dec 19, 2024

jsitarek Dec 19, 2024

jsitarek Dec 19, 2024

jsitarek Dec 19, 2024

jsitarek Dec 19, 2024

jsitarek Dec 19, 2024

jsitarek Dec 19, 2024

jsitarek Dec 19, 2024

jsitarek Dec 19, 2024

jsitarek Dec 19, 2024

jsitarek Dec 19, 2024

jsitarek Dec 19, 2024

jsitarek Dec 19, 2024

		@@ -0,0 +1,7 @@
		timestamp,base_height,top_height,transmission,lidar_zenith

		@@ -280,8 +280,8 @@ dl2_to_dl3:

		cloud_correction:
		specify_cloud: yes


		subarray_info.to_hdf(output_file)

		correction_params = config.get("cloud_correction", {})

Adding a function to create a log file and save cloud correction para… #260

Are you sure you want to change the base?

Adding a function to create a log file and save cloud correction para… #260

Conversation

nzywucka commented Sep 13, 2024

codecov bot commented Sep 13, 2024 • edited Loading

Codecov Report

jsitarek commented Sep 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsitarek commented Oct 2, 2024

jsitarek commented Oct 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Sep 13, 2024 •

edited

Loading