[QUESTION] How to release the model and optimizer memory manually?

[robotsp]

Your question
How to release the model and optimizer memory manually?

What I have tried

• set zero_grad() method
• set None method
• del method
• gc.collect()
• torch.cuda.empty_cached()

But the ways above are not worked, please help, thanks!

[deepakn94]

What's the use case for releasing the model memory? Trying to delete the optimizer object might help with releasing the optimizer memory (so something like del optimizer in megatron/training.py).

[robotsp]

What's the use case for releasing the model memory? Trying to delete the optimizer object might help with releasing the optimizer memory (so something like del optimizer in megatron/training.py).

I have a use case to call the training function twice, but I found some memory fragment that cannot release by del and torch.cuda.empty_cache() after the first dumpy training. @deepakn94

[deepakn94]

I see, makes sense. You probably don't want to release model memory still. Presumably you are not doing a backward pass the first time you run the training function?

You could try activation recomputation.

[robotsp]

Actually, the use case is required to change the original neural network structure and that's why I want to release the model and optimizer memory from the original one. I tried recomputation to release activation in the dummy training already. But model and optimizer should occupy more memory, I think?

I have clear model parameters and optimizer state by setting tensor.storage.resize_(0). It did drop some of the memory fragment but not that much.

@deepakn94 Thanks for your quick answer in advance!

[deepakn94]

Hmm, not sure. del model and del optimizer seem like the "right" things to do here.

[robotsp]

@deepakn94 One more question, I found I trained the model in the second time after the dummy first training, its loss curve is different from that one in the training from scratch. Do you know why? Should I destroy the process group and reinitialize it?

[github-actions[bot]]

Marking as stale. No activity in 60 days.