'ColossalAI/examples/language/gpt' and 'ColossalAI-Examples/language/gpt' which is the latest demo? #3388
Closed
xinghua3632
started this conversation in
Community | General
'ColossalAI/examples/language/gpt' and 'ColossalAI-Examples/language/gpt' which is the latest demo?
#3388
Replies: 1 comment
-
|
TKS, I have a new question |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
ColoInitContext is more recommended than ZeroInitContext to build model?
In 'ColossalAI/examples/language/gpt', demo in gemini and three demos in experiments which is your recommended? I just want to use all solutions in Colossa for my projects, data parallel, model parallel, and pipeline parallel;
When I set TPDEGREE=2 in run_gemini.sh in 'ColossalAI/examples/language/gpt', the usage of GPU Memory is bigger than TPDEGREE=1, meanwhile the model cannot load pretrained parameters directly ,because the tensons in model are splited to half, but tensors in checkpoint are whole tensors.
Do you have a completed and latest demo that use all parallel mode?
Beta Was this translation helpful? Give feedback.
All reactions