Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable Append Mode in SaverIterDataPipe #1270

Open
rravu3 opened this issue Jun 10, 2024 · 1 comment
Open

Enable Append Mode in SaverIterDataPipe #1270

rravu3 opened this issue Jun 10, 2024 · 1 comment
Labels
datapipes wontfix This will not be worked on

Comments

@rravu3
Copy link

rravu3 commented Jun 10, 2024

🚀 The feature

Currently, Saver only allows write mode and only users to choose byte vs text mode. It might be useful to allow the flexibility to append to an existing file.

Motivation, pitch

Allowing the flexibility to append could be useful for building data pipes that can enable large scale transformations on original dataset like tokenizing, randomizing and/or splitting across training and evaluation.

Alternatives

No response

Additional context

No response

@andrewkho
Copy link
Contributor

Thanks for the feature request @rravu3 , unfortunately we will be deprecating and then deleting DataPipes/DataLoaderV2, please see this issue: Future of torchdata and dataloading

@andrewkho andrewkho added wontfix This will not be worked on datapipes labels Jun 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datapipes wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

2 participants