Memory leak

[amjltc295]

I notice that there seems to be a memory leak issue during training.

On my K80, I set the batch size to 24 and the GPU memory consumption is about 4000MB. However, as the training process goes on, the GPU memory consumptions increases and a RuntimeError: CUDA error: out of memory is raised about 30 minutes later. If I set the batch size to 16 the error would not take place but the memory still increase.

[YupengGao]

@amjltc295 I have the same issue RuntimeError: CUDA error: out of memory. Have you solve the problem yet?
or do you have any idea how to debug this kind of error?

[amjltc295]

@YupengGao No, I just keep the batch size small. I wish someone could give me a clue.
For debugging, https://discuss.pytorch.org/t/how-to-debug-causes-of-gpu-memory-leaks/6741/3 might help