Fix DIPOL bugs, add quad tests, improve parallelization, other improvements #39
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
juanep97
changed the title
improve parallell reduction by setting synchronous = NORMAL journal_mode = WAL
PhotoPolResult.create: pass clear=True to photopolresult.reducedfits.set
include handling of BLLac special case
|
If I can suggest something for the future, to easier track changes, it is better to solve one bug at a time in single PRs |
I know, sorry. The pace at which we have made changes lately to have a functional pipeline for the latest campaigns has forced us to work at the same time on many issues. |
morcuended
reviewed
Nov 24, 2023
morcuended
approved these changes
Nov 27, 2023
previous behavior was equivalent to chmod 'a=r', now 'a-w'.
juanep97
changed the title
juanep97
changed the title
the func modifies the value so subsuquent iterations had different start args it was making some files fail in bulk reduction but not when processed alone in general, it is not recommended to have mutable types as default args
now trying to use missing Instrument definitions will give error
morcuended
reviewed
Nov 29, 2023
it has not been used in along time and it is not mantained it is not really needed after the last improvements in parallelization
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Solves #22 , #38, and some bugs introduced in #23.
Summary of changes:
max_concurrent_threads > 1, it reduces all polarimetry groups for each epoch in parallel. This reduces time a lot especially in nights with DIPOL data.Concurrency has been improved greatly. Previously there was some isolated 'database errors' with ~10 cores, and many more with 12-14. Now I have tested with up to 16 cores and there are no errors, probably even more cores are fine too.
This is a nice point to create a release (#37).