Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Split large ancestor groups up for both caching and dask scheduling. #838

Closed
benjeffery opened this issue Jun 18, 2023 · 1 comment
Closed

Split large ancestor groups up for both caching and dask scheduling. #838

benjeffery opened this issue Jun 18, 2023 · 1 comment
Milestone
Release 0.4.0

Comments

@benjeffery
Copy link
Member

We have very large ancestor groups towards the end of matching. As these take over a month of CPU each it would be best to split them up. This would mean we could resume them mid-way and have smaller dask.bag partitions. The smaller partitions result in better utilization from interrupting tasks due to worker roll-over and also reduce wasted worker time at the end of a group due to better tesselation of smaller tasks.
@benjeffery benjeffery added this to the Release 0.4.0 milestone Jun 18, 2023
@benjeffery
Copy link
Member Author

Superseded by #921

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Release 0.4.0
Development

No branches or pull requests
1 participant
@benjeffery