Distributed training with Horovod?

[Qiuzhuang]

Can we have the Distributed training with Horovod version -- we want to speed up LM training via a cluster of GPU machines.

[raulpuric]

We do not use horovod for pytorch distributed training. We use our own multinode launchers that adhere to pytorch's distributed training format for multinode training.
Pytorch's multinode format is mpi-like and fairly similar to horovod so you may be able to use it, but I can't say for sure.
Here's a more updated example of PyTorch distributed training for language modeling. You should be able to set similar environment variables to run multinode training.