Do ZeRO2 and ZeRO3 need gradient accumulation?

[Gy-Lu]

As we know, ZeRO2 and ZeRO3 would split the gradient, which is incompatible with gradient accumulation.
However, they are not that incompatible.
For instance, we can accumulate the gradients belonging to each rank after communication.

Drawback

This version of gradient accumulation saves no communication.

Advantage

For some users who need a large batch to train their model(for model convergence, e.g.) but have limited GPU memory, gradient accumulation can be a solution.

[Gy-Lu]

@ver217 @FrankLeeeee @kurisusnowdeng