-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow partial batches #177
base: main
Are you sure you want to change the base?
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #177 +/- ##
==========================================
+ Coverage 96.25% 96.30% +0.05%
==========================================
Files 6 6
Lines 347 352 +5
Branches 82 83 +1
==========================================
+ Hits 334 339 +5
Misses 8 8
Partials 5 5 ☔ View full report in Codecov by Sentry. |
@arbennett sorry for the very slow response! This is great! It would be good to include add a test for this functionality before merging. Please let me know if you'd want to add a test or I can push one if you don't have time. |
@maxrjones @arbennett I wondered if we have any updates on this PR. I recently used xbatcher in a tutorial delivered by Pangeo members in Europe. I added a warning note indicating xbatcher doesn't (yet) return partial batches and mentioned some workarounds (please see the note before the References section). @weiji14 you've suggested posting this somewhere. I hope this is the correct place for starting the conversation and follow-up actions. |
@acocac - thanks for pinging me, I totally lost track of this. I just merged upstream changes into my branch. @maxrjones - sorry I also missed your ping here, I'll see if I can add a test for this real quick. |
@maxrjones - okay test added! Sorry for the bad commit message, my backticks got the best of me 🤦 Anyhow, I think the test is implemented correctly, but the need to do the rounding is kinda ugly IMO. Let me know if you have an easy way to clean that up. |
Thanks @arbennett! Mind if I push to your branch to fix the pre-commit errors? I'll also take a look at the rounding logic today |
Description of proposed changes
This is a simple additional flag that allows for "partial" batches to be returned.
Fixes #123