Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow partial batches #177

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

arbennett
Copy link

Description of proposed changes

This is a simple additional flag that allows for "partial" batches to be returned.

Fixes #123

@codecov
Copy link

codecov bot commented Mar 21, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.30%. Comparing base (43c9135) to head (8b28973).
Report is 11 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #177      +/-   ##
==========================================
+ Coverage   96.25%   96.30%   +0.05%     
==========================================
  Files           6        6              
  Lines         347      352       +5     
  Branches       82       83       +1     
==========================================
+ Hits          334      339       +5     
  Misses          8        8              
  Partials        5        5              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@maxrjones
Copy link
Member

@arbennett sorry for the very slow response! This is great!

It would be good to include add a test for this functionality before merging. Please let me know if you'd want to add a test or I can push one if you don't have time.

@acocac
Copy link

acocac commented Jul 8, 2024

@maxrjones @arbennett I wondered if we have any updates on this PR. I recently used xbatcher in a tutorial delivered by Pangeo members in Europe. I added a warning note indicating xbatcher doesn't (yet) return partial batches and mentioned some workarounds (please see the note before the References section).

@weiji14 you've suggested posting this somewhere. I hope this is the correct place for starting the conversation and follow-up actions.

@arbennett
Copy link
Author

@acocac - thanks for pinging me, I totally lost track of this. I just merged upstream changes into my branch.

@maxrjones - sorry I also missed your ping here, I'll see if I can add a test for this real quick.

@arbennett
Copy link
Author

@maxrjones - okay test added! Sorry for the bad commit message, my backticks got the best of me 🤦

Anyhow, I think the test is implemented correctly, but the need to do the rounding is kinda ugly IMO. Let me know if you have an easy way to clean that up.

@maxrjones
Copy link
Member

Thanks @arbennett! Mind if I push to your branch to fix the pre-commit errors? I'll also take a look at the rounding logic today

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Provide the option to return partial batches
3 participants