Memory error while training

[hmchuong]

I used 2xTeslaK80(12GB each) for training from scratch with batchSize=16 but always got out of memory at the end of the first epoch. Have you trained with the same batchSize on your machine?

[QinyuHu]

I feel like it is because of the way he passed loss to scheduler. Instead of only adding values during develop, he added tensor, which accumulated too many memory. You can just unpack the value.
losstensorMean += losstensor
to
losstensorMean += losstensor.data[0]

[hmchuong]

when we change to that, the losstensorMean and lossVal will receive the same value, isn't it?

[taijizhao]

@hmchuong hi hmchuong, I trained on 2 1080ti and also met the problem. Meanwhile, the GPU memory usage isn't too high. Did you solve the issue? Thanks!

[yan-michael]

The problem is still ongoing, any suggestions?? Adding [0] or .item() does not work either, even for batch size of 2.