Load an AWQ model via python API

[GhostXu11]

Hi, I recently wanted to use litserve to deploy a service and use litgpt to load the model. However, due to video memory reasons, I wanted to use an AWQ model. However, I did not see the method of loading the awq model in the python API document. Do you have any methods?

[Andrei-Aksionov]

If you want to quantize an original model to lower the VRAM consumption, you can use BitsandBytes quantization that does it on-the-fly: https://github.com/Lightning-AI/litgpt/blob/main/tutorials/quantize.md

But if you have weights in AWQ format and want to load them to LItGPT - it's not supported.
There was an attempt to support AutoGPTQ (that should also support AWQ format) in #924, but it was never merged.
(Making that PR up-to-date might could be a cool contribution 😉).

[GhostXu11]

thanks for your reply :)