How to use multiple GPUs to finetune the model?

[cchangyou]

Hi, if I follow the instruction to run image_train_latent.py, it seems only one GPU is used. Can you advise on how to use multiple GPUs? Thanks.

[limiteinductive]

You can use the mpiexec -n N python command as detailed in this repo: https://github.com/openai/guided-diffusion

[cchangyou]

Got it. Thank you.

[alishan2040]

@cchangyou Did mpiexec -n N python work in your case? because I tried to use it with multiple gpus but still facing memory error as it is only using GPU 0.

I used: mpiexec -n 4 python

Thanks.

[limiteinductive]

@alishan2040 The load is not shared among GPUs, you'll need multiple GPUs with enough VRAM each

[alishan2040]

@limiteinductive How much VRAM should be considered enough for a single GPU? Now I've 4 gpus with 16 GB VRAM each. Previously I had single GPU with 24 GBs VRAM. In both the cases, I faced memory errors.

[limiteinductive]

@alishan2040 I tried only using A100's