ZTF alerts aux: reprocessing scripts #259
Conversation
Theodlz
changed the title
| return doc, prv_candidates, fp_hists | ||
|
|
||
|
|
||
| def process_file(argument_list: Sequence): |
There was a problem hiding this comment.
Should these go in dedicated functions?
There was a problem hiding this comment.
Hi @mcoughlin. All of these methods already exist but as class methods (which is what they should really be for nightly processing). Of course, it'd be better if they were in a spot where both the class and a script could use them, but I'd rather not modify the nightly processing class, and have a duplicate here, given that this script will basically run just once, and then be removed from the repo.
Basically, this is for me to try and see if a script like this (with one thread per day of alerts being processed) is quicker than N threads that all submit to a dask cluster. If they turn out to be much quicker, then I'll leave that script here and find a cleaner solution to avoid the duplication.
This PR ads 2 scripts that are useful if one wants to reprocess lots of alerts, to reconstruct the full light curves found in the ZTF_alerts_aux table.
We already have a branch with code to do that from the Kafka alert stream, but this allows you to do the same using the tarballs available online or any set of alert files.