Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize to_hipscat using dask delayed #64

Merged
merged 4 commits into from
Nov 14, 2023

Conversation

smcguire-cmu
Copy link
Contributor

Changes the write_partitions method to use dask_delayed to write out partitions in parallel in the dask task graph. This is how dask writes to disk.

Copy link

codecov bot commented Nov 13, 2023

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (76e6411) 99.13% compared to head (61b8890) 98.94%.
Report is 6 commits behind head on main.

Files Patch % Lines
src/lsdb/io/to_hipscat.py 95.23% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #64      +/-   ##
==========================================
- Coverage   99.13%   98.94%   -0.20%     
==========================================
  Files          21       21              
  Lines         464      473       +9     
==========================================
+ Hits          460      468       +8     
- Misses          4        5       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

github-actions bot commented Nov 13, 2023

Before [d57f770] After [6f97cf3] Ratio Benchmark (Parameter)
34.2±0.6ms 33.7±0.2ms 0.99 benchmarks.time_kdtree_crossmatch

Click here to view all benchmarks.

src/lsdb/io/to_hipscat.py Outdated Show resolved Hide resolved
src/lsdb/io/to_hipscat.py Outdated Show resolved Hide resolved
@smcguire-cmu smcguire-cmu merged commit 49a169b into main Nov 14, 2023
12 of 14 checks passed
@smcguire-cmu smcguire-cmu deleted the sean/parallelize-catalog-saving branch November 14, 2023 23:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants