Out Of Memory Error

[bianxuxuxu]

When I use quantize_ function and run the model, then comment it out in next running time, the original model will appear OOM, and it seems that the pytorch GPU memory automatic reuse function is no longer available. As the running continues the memory to grow until OOM.(My model contains a stack of some modules)

mymodel = MyModel().eval().cuda().half()
quantize_(mymodel, int4_weight_only()) #then comment it in next running

#gen random input
input_ = torch.rand ...
#run model forward
output_ = mymodel(input_)
print(output_)

Is there any cache config be saved in the torchao quantize process and it couldn't be clear?

[jerryzh168]

Hi @bianxuxuxu can you give a repro so we can debug the issue?

[bianxuxuxu]

Hi, @jerryzh168
There may be something wrong with my environment, I'll check again. Thank you