Issue #311 create cl make bin stage

[marina-ricci]

Linked to #311. This issue creates a stage to bin the cluster catalogs in bins of richness and redshift and output a tomography TX file. Pipeline and config files are also written to run the CL pipeline, as well as the batch script for CCin2p3 and a notebook running the pipeline and exploring the input and output files.

[marina-ricci]

• binning stage : class CLClusterBinningRedshiftRichness in
  bin cluster.py

• Pipeline to run the CL pipeline:
  pipeline-20deg2-CL-in2p3.yml
  pipeline-1deg2-CL-in2p3.yml
  pipeline-20deg2-CL-nersc.yml
  pipeline-1deg2-CL-nersc.yml

• config files for the CL pipeline :
  config-20deg2-CL.yml
  config-1deg2-CL.yml

• batch script for CCin2p3:
  20deg2-in2p3.sub
  1deg2-in2p3.sub

• batch script for NERSC:
  20deg2-nersc.sub
  1deg2-nersc.sub

• notebook running the pipeline and exploring the input and output files : (some paths to change, or lines to comment/uncomment depending on if you are running on NERSC vs. IN2P3)
  Run_CL_pipeline.ipynb

For the CLClusterBinningRedshiftRichness stage :

# This is the initial cluster catalog - RAs, Decs, richess, redshift, etc.
cluster_catalog="./data/example/inputs/cluster_catalog.hdf5",

# This is the output for this stage
cluster_tomography="./data/cosmodc2/outputs-1deg2-CL/cluster_catalog_tomography.hdf5",

I explicitly wrote different bin definition in the config file and as the default in the pipeline stage to make it explicit in the notebook which is what.

[marina-ricci]

@cavestruz : the PR is ready on my side. Normally you should have all info from the notebook! Please test the nersc launcher (just for the 1deg2 should be sufficient) as I did not test it. We should find better way to work at the two site with changing just the launcher file, but I did not find the proper way to do it. So you also have to change the first lines in the pipeline files.

[cavestruz]

Primary changes are to change the 20sqdeg yaml files so this works.

[cavestruz]

Testing this submission in NERSC.

sbatch examples/cosmodc2/1deg2-nersc.sub

[cavestruz]

I copied this file to two separate config-1deg2-CL-{in2p3,nersc}.yml and put each option in the notebook, Run_CL_pipeline.ipynb so the comment/uncommenting happens at the notebook level as opposed to inside the yml file.

[cavestruz]

• This has some lines that need updating to run, consistent with the 1deg2 version.

[cavestruz]

• This has some lines that need updating to run, consistent with the 1deg2 version.

[cavestruz]

I made some minor additions here to account for different in2p3 vs. nersc files.

[cavestruz]

Made minor change here, so there are separate yaml files depending on where the user is running the notebook.

[cavestruz]

Testing this out now with the squeue submission. Will then check if latter portion of notebook runs on NERSC with the output of the 20deg2 submission that uses this yaml file.

[marina-ricci]

@cavestruz : I did some cleaning as there were typos in the files and reorganized the notebook to clarify the 2 approaches to run : direclty from jupyter or via pipeline/ceci. I commented all unused stages / input so that the pipeline chart for this PR is just:

Please check again everything runs, it should be very fast, even on the 20deg2.

[cavestruz]

Great - I am able to run everything in the notebook, and submit the sbatch examples/cosmodc2/20deg2-nersc.sub to run the tail end of cells in Run_CL_counts_pipeline.ipynb.

[cavestruz]

Testing this out now with the squeue submission. Will then check if latter portion of notebook runs on NERSC with the output of the 20deg2 submission that uses this yaml file.