Any existing Alpaca models out there?

[JustinHinh]

I'm struggling to find the latest Alpaca model that was given additional training via Divinci 3

I have an RTX 3080 10gb and have no idea if it's even strong enough to run it

Any help is appreciated

[BetaDoggo]

The only version I know of that was made with native training is alpaca-native trained by dep on the SAIL discord. There are some others that were trained with LORA going up to 30B but they don't follow instructions as well from my experience. You can definitely run it with 10GB but you may need to convert the weights to 4bit or offload to ram.

[madmads11]

I have been running an Alpaca LORA 30b model on my CPU with llama.cpp quite successfully, I wouldn't discount the LORA models entirely. It's definitely not perfect though, agreed.

[madmads11]

Does anyone know of a way to run the Alpaca LORA 30b models on GPU with text-generation-webui? I'm thinking this model specifically: https://huggingface.co/elinas/alpaca-30b-lora-int4 (or any other Alpaca 30b int4 models that may come along).

Since it's basically just a fine-tuned LLaMA 30b model, does that mean I can load it in the same procedure with text-generation-webui as I would the LLaMA models? I would love to try it for myself, but I don't have the hardware to do it locally and I want to hear if anybody knows if it's possible before I look into running it in the cloud.

[BetaDoggo]

The best way to do this for now is probably to merge the lora into the model then quantize it to 4bit.

I haven't actually gone through this whole process with 30B or newest GPTQ so I can't confirm that it currently works, but based on the documentation this should be the general process:

1. Merge the lora into the model using export_hf_checkpoint.py in the alpaca-lora repo by replacing decapoda-research/llama-7b-hf with decapoda-research/llama-30b-hf(or your own hf format 30b) and tloen/alpaca-lora-7b with the 30b lora(chansung/alpaca-lora-30b most likely).

2. You can then quantize the converted model using the GPTQ-for-LLaMa repo by running CUDA_VISIBLE_DEVICES=0 python llama.py <model path> c4 --wbits 4 --groupsize 128 --save llama30b-4bit-128g.pt

Notes:
I've heard that the gptq-for-llama repo has had some recent changes that broke things in the webui and required gptq models to be regenerated, so make sure that your webui is using the newest version. I'm also not sure how well group size is supported(last I heard it wasn't but idk), so if the model doesn't work you should try removing that argument.

If you end up following this and getting it to work, please post any changes or extra steps you had to take. I'd be happy to help if there are any errors during the process. I've yet to find any up to date guide on this so I'd like to have a known working method.

Edit: The model you linked should work, you just need to place the .pt file in the root of the models folder then remove the "-int4" from the original folder name.

[madmads11]

Thank you so much for the reply and details! Hopefully it can inspire someone to pick up the project because unfortunately, it is a little more work than I am able to commit right now. However, I would also follow the efforts closely if anybody wanted to attempt this themselves.