Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tokenization crashes when using deepspeed #117

Closed
dlwh opened this issue Mar 10, 2022 · 0 comments
Closed

Tokenization crashes when using deepspeed #117

dlwh opened this issue Mar 10, 2022 · 0 comments
Labels
bug Something isn't working
Milestone
Mistral V2

Comments

@dlwh
Copy link
Member

dlwh commented Mar 10, 2022

Describe the bug

via @J38 in #109 :" tokenization crashes when using deepspeed"

We should either refactor entry point to run data processing and then call launcher with subprocess after data has been processed

Or, better, stream data #116

To Reproduce
Steps to reproduce the bug. TODO

Expected behavior
don't crash

Additional context

related to #116
@dlwh dlwh added this to the Mistral V2 milestone Mar 10, 2022
@dlwh dlwh added the bug Something isn't working label Mar 10, 2022
@dlwh dlwh mentioned this issue May 31, 2022
Merged
@dlwh dlwh closed this as completed Jun 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
Mistral V2
Development

No branches or pull requests
1 participant
@dlwh