The GPTQ startup delay has been fixed lads.

[USBhost]

Please go update https://github.com/qwopqwop200/GPTQ-for-LLaMa and have fun
65b 4bit used it be 50s plus now look how fast it is

[Ph0rk0z]

What was the fix?

[USBhost]

qwopqwop200/GPTQ-for-LLaMa@c589c54

[Ph0rk0z]

That sped it up for me but delay still seems greater than using https://github.com/johnsmith0031/alpaca_lora_4bit

edit: ok I ran more benchmarks and it's about 95% the same

[OWKenobi]

Amazing! It also speeds up the llama-30b-4bit-128g. Before the patch, the delay was 11 seconds. After the patch, the delay is 3 seconds!!

BTW, how do you run the 65bit model? You happen to have two 3090s?

[USBhost]

RTX A6000

[OWKenobi]

Nice! Secretly I had hoped you had discovered a dirty hack to make it run on a 3090. On the other hand, at the speed of development these days this means I will have to wait roughly 2 more weeks till I can run it ^^

[nullgate]

RTX A6000

@USBhost I have RTX 3090 and I'm planning to buy A6000 or 6000 ada. I would expect A6000 slower than RTX 3090 on LLaMA-30B. Can you tell about your experience with this A6000?

[USBhost]

RTX A6000

@USBhost I have RTX 3090 and I'm planning to buy A6000 or 6000 ada. I would expect A6000 slower than RTX 3090 on LLaMA-30B. Can you tell about your experience with this A6000?

Sorry for the delay. I have no experience with the 3090 All I know is when it comes to AI it's a little faster than one. The A6000 is max a 300w card. Other than that amazing experience. You don't even hear the blower. The card is set to try and not go over 86c. But even at 100 fan speed it's quiet.

[bbecausereasonss]

I'm still super confused as to which GPTQ branch I need to be using. Seems many of the models are coming out based on the new one, but should we be using the old branch?

[Ph0rk0z]

You need triton for act order + group size together I think. If your models don't use that, its fine to use the oobabooga fork or any other v2 one.