Preprocess: split multiple process pipes pre/post demod

[msilvafe]

We need the ability to run multiple preprocess pipelines that each have unique config files and databases and are dependent on one another. The main use case is splitting the SAT preprocessing into pre- and post-demodulation pipelines. The idea being that all cuts, flags, and filters pre-demod will most likely live 100% inside of the flp/site-pipeline framework however there are multiple ideas for filtering and mapmaking after demodulation which would require different post-demod pipelines (or not use the preprocessing module at all post-demod). The tricky points that will need some development/infrastructure work I can see with this are:

1. The driving scripts are either load_preprocess_tod which pass a proc_aman to pipe.run or preprocess_tod which passes no proc_aman so there's actually no way to have the output axis manager of one load_preprocess_tod or preprocess_tod to be passed to a new call of pipe.run without any proc_aman we need a new driving script that takes in two config files with unique databases and writes to the second database (dependent on the processing in the first).
2. How to keep track of the multiple pipeline dependencies i.e. post-demod pipelines a, b, c are dependent on pre-demod pipeline d
3. How to keep flags that are computed in post-demod steps but when only loading in processing up through demod.

[iparask]

2. How to keep track of the multiple pipeline dependencies i.e. post-demod pipelines a, b, c are dependent on pre-demod pipeline d

You can either use a workflow management system that provides the capabilities to define dependencies, or use mpi4py futures to create sid dependencies.

[msilvafe]

2. How to keep track of the multiple pipeline dependencies i.e. post-demod pipelines a, b, c are dependent on pre-demod pipeline d

You can either use a workflow management system that provides the capabilities to define dependencies, or use mpi4py futures to create sid dependencies.

Thanks @iparask! I don't think I need mpi so I'll skip mpi4py for now but can you give me an example of a suitable workflow management system?

[msilvafe]

@mmccrackan and I decided that he will write a new site_pipeline script called multilayer_preprocess that will handle:

1. Taking in 2 cfg files and writing 2 databases successively
2. Taking in 2 cfg files and loading from 1 and writing a 2nd
3. Loading in/processing data through multiple layers of preprocess cfgs/dbs

I will write the config, metadata, and database order copying functions that will be embedded across all preprocess related site_pipeline scripts and the config checking functions to be used in multilayer_preprocess