how to train a bert model with distributed training ?

[MarvinLong]

No description provided.

[MarvinLong]

I saw a bert is trained with 64 GPUs in 3.3 days in this article
https://medium.com/future-vision/bert-meets-gpus-403d3fbed848?fbclid=IwAR0bFskUVVKDRyYF-9cQGgRXeq7dTvteGHi10HaTG5zI7_eE8oW-BfrxYQw

I want to know is this repo or the pytorch version in https://github.com/NVIDIA/Megatron-LM ?
Can you help me to train a bert model with distributed training, thanks.

[swethmandava]

We currently only published scripts for single node training. Stay tuned for distributed multi node training scripts, we will publish them soon.

[LifeIsStrange]

@swethmandava
Why doesn't Megatron allow us to open issues?
For example it would be nice if it supported
https://github.com/zihangdai/xlnet
Which is the new state of the art (consistently beat BERT) as you can see on paperswithcode.com
And it does not yet support multi gpu zihangdai/xlnet#218

(it would be nice to support ERNIE 2.0 too but less of a priority)

[LifeIsStrange]

BTW nvidia is already contributing to xlnet e.g this nvidia employee:
zihangdai/xlnet#200
So let's be consistent

[swethmandava]

Multi node training is now supported from #208

@LifeIsStrange you can also open issues on Megatron-LM now, thanks for contributing!