Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces a bundle of changes, that, together with #396, reduce the time to initialize an injector and run a single trial with 10k sources and the 10-year NT dataset (gamma_precision reduced to 0.25, so only 8 gamma points) from 6.2 hours to 3 minutes. Of that, a solid 2 minutes are spent in
load_data()
*. The speedups arise as follows:astropy.table.Table
, sorted by declination, rather than in a gigantic structured array. This helps in two ways. First, operations on individual columns of the table make better use of caches, since the values are stored contiguously in memory, rather than 176 bytes apart as in the array. Second, sorting the table by declination means that the expensive band-masking step can be replaced by a cheap binary search. This is also more cache-efficient than indexing with a boolean array, since applying a boolean mask requires reading every single value in the column (or worse, the structured array). This injector is in all ways better than LowMemoryInjector (and perhaps, the base MCInjector)*Most of the remaining fat is in flarestack.icecube_utils.dataset_loader.load_dataset() and its use of append_records(). This could be obviated by using astropy Table everywhere, but would make the most sense if the data were stored that way in the first place. That's probably too large of a change at this point.