Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow disabling inlining for MultiZarrToZarr #506

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

maresb
Copy link

@maresb maresb commented Sep 21, 2024

I am trying to combine some massive S3-based Zarr references. A MultiZarrToZarr operation which would otherwise take a minute would take days because the size of each file on S3 is being queried. Thus I want to be able to pass in inline_threshold=None so that my datasets combine in a reasonable timeframe.

@martindurant
Copy link
Member

I think this was already possible with inline_threashold=0 ?

@maresb
Copy link
Author

maresb commented Sep 24, 2024

Nope, if inline_threshold=0 then fs.info(fn) still gets called in order to ensure that the file size isn't negative. 😅

We could change this so that we use 0 in place of None. That would probably make more sense actually. Then my conditional self.inline is not None becomes self.inline > 0.

@martindurant
Copy link
Member

That sounds like a good plan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants