You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
With dask-awkward and integration with hist, it seems conceptually possible to make the template histogram creation fully Dask-driven. This could result in two significant improvements:
parallelization of template histogram construction,
optimization of data processing, e.g. avoiding duplicate data reading when filling lots of histograms that only differ by a weight.
The latter requires dask-awkward and dask-histogram and needs to build the full task graph before calling compute. This is a much bigger change than providing an interface that allows distribution of template construction (which acts as a black box function) via e.g. Dask.
The text was updated successfully, but these errors were encountered:
An intermediate solution that might also help is the possibility to run histogram production on a subset of all templates via some filter, allowing to parallelize manually by calling the same function with different filters.
With
dask-awkward
and integration withhist
, it seems conceptually possible to make the template histogram creation fully Dask-driven. This could result in two significant improvements:The latter requires
dask-awkward
anddask-histogram
and needs to build the full task graph before callingcompute
. This is a much bigger change than providing an interface that allows distribution of template construction (which acts as a black box function) via e.g. Dask.The text was updated successfully, but these errors were encountered: