New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

save checkpoint with error #50

Open

13613979212 opened this issue Aug 28, 2024 · 1 comment

13613979212 commented Aug 28, 2024

set max_steps=500, save_steps=100
When it reaches step 100, the checkpoint is saved successfully but nccl_timeout is displayed

Collaborator

Muennighoff commented Aug 28, 2024

i think that is the first issue mentioned here: https://github.com/ContextualAI/gritlm?tab=readme-ov-file#known-issues - don't have a better solution atm than what is said there

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment