LST AGN Analysis

This project tries to handle all the steps needed for a typical AGN-analysis (or any point-source really):

• Select runs observing the source and matching some quality criteria
• Apply models on dl1 files / Produce dl2
• Calculate IRFs
• Produce dl3
• Run the gammapy data reduction to dl4
• Perform spectral fit, calculate flux points, calculate light-curve, ...
• Perform multiple gammapy analyses with the same dl3

What it does NOT handle:

• Low-level calibrations. We start at DL1

Conceptually there are three main steps (You could define more, but these are still somewhat split in the workflow):

• Select runs via the datacheck files
• Link runs and mc to the build directory. This makes rules easier to reason about. These are only links, not copies (!) and show up as such using ls -l.
• Run the analysis using lstchain and gammapy

The different workflow stages are separated into different files where I saw a clear distinction. This is to keep each file relatively small and reduce the number of variables one has to reason about at once. The different stages are not directly usable on their own as they assume a certain directory structure and file naming conventions, but it should be straightforward to adapt this to ones liking if one would want to build their own analysis. I try to keep all variables in workflow/definitions.smk, which should cover most, but not necessarily all. Also the scripts should all be usable on their own with the small caveat, that sometimes things get imported from the top level scripts directory, so you will need to make sure these things are present as well or copy them there.

Usage

Prerequisites

You need configuration files. Check our example repository for how to use them, and clone them next to this repo, e.g.

git clone https://github.com/nbiederbeck/lst-analysis-config.git ../lst-analysis-config

You need snakemake and astropy installed. If you do not have that, you can create an enviroment with only these packages like this:

mamba env create -f workflow/envs/snakemake.yml
mamba activate snakemake

For development, you should also install atleast pre-commit. The enviroment in workflow/envs/data-selection should contain everything (Don't forget to pre-commit install afterwards). This is not a specific development enviroment though. Cleaning up the envs is definetively a DEV-TODO as it got a bit messy after merging what was originally two projects.

Also you need the source catalogue. Since this requires credentials (standard LST ones), it is not done automatically. This is needed only once (or whenever you want to use new runs). You can use this command (replace <username> and <password>):

curl --user <username>:<password> \
    https://lst1.iac.es/datacheck/lstosa/LST_source_catalog.html \
    -o runlist.html

Config

Adapt to your liking

• Data selection: ../lst-analysis-config/data-selection.json
• MCs to use, lstchain env and number of off regions: ../lst-analysis-config/lst_agn.json (gammapy does not handle energy-dependent cuts automatically, so we need to work around this)
• IRFs (lstchain): ../lst-analysis-config/irf_tool_config.json
• gammapy: analysis.yaml and models.yaml in subdirectories ../lst-analysis-config/analysis_*. These all use the same dl3, but produce their own dl4

If you want to use configs from a different place, instead call make using make CONFIG_DIR=/path/to/configs.

Run the analysis

Basically only run make. If you run it without a specific target, everything should be resolved. In the event, that you give a target such as make build/plots/analysis-one/flux_points.pdf, the linking part might not run again. This is clearly suboptimal, but for now you will need to run make link first IF YOU CHANGED SOMETHING in the data selection. Since in this case, one probably wants to rerun everything anyway, this was not super high on our priority. It is related to nbiederbeck#26

Local Usage

If you have run snakemake on the cluster, you can create the plots and tex files locally (using your own matplotlibrc for example). We separate the calculation of metrics and the plotting to make sure you can finetune plots later on without needing to run the expensive steps on the local machine. The tables for that are saved as either fits.gz or h5.

For the data-selection plots, you need to download build/dl1-datacheck-masked.h5, e.g.:

mkdir -p build
scp <host>:<path-to>/lst-data-selection/build/dl1-datachecks-masked.h5 build/

Afterwards:

make -f local.mk

DEV-TODO: Do the same for the other plots (nbiederbeck#29) If you do some cp **/*.fits.gz shenanigans, beware that the dl3 files are saved with the extension fits.gz as well.