This repository has been archived by the owner on Mar 23, 2023. It is now read-only.

grad is none when run gpt2 with pipeline parallelism only #204

Open

lin88lin8850 opened this issue Dec 22, 2022 · 0 comments

lin88lin8850 commented Dec 22, 2022

🐛 Describe the bug

Environment

torch==1.12.0a0+8a1a93a
num_gpu=4
pipeline=4
model = gpt2-small

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.