Questions about GPU requirements and usage

[SlenderMongoose]

Thank you for sharing your code and detailed documentation. I am currently using your code and have encountered a few questions regarding the GPU usage.

1. What is the approximate GPU memory requirement for fine-tuning？
2. Is it possible to train the model using a single GPU? If so, what settings should I be aware of?
3. I am currently using multiple RTX 3090 GPUs but encountering out-of-memory (OOM) errors. Is there any way to resolve this issue? If so, how should I set it up?
4. Interestingly, I can successfully run test.py using a single RTX 3090 GPU, but it fails with multiple RTX 3090 GPUs. Do you have any suggestions on how to resolve this?

Thank you very much for your time and assistance!

[zhengbw0324]

@SlenderMongoose
Hello, it should be possible to train our model on 3090. You can try the following strategies:

1. Adjust the batch size.
2. Try to enable deepspeed offload, you can refer to here.
3. Try to use LoRA for fine-tuning.

However, we do not recommend that you use 3090 for experiments, as it will result in unacceptable time consumption.

[zhengbw0324]

As for the failure of running the test with multiple GPUs, I think it may be because the communication between multiple GPUs takes up extra memory. You can reduce the batch size or num_beams appropriately.