Testing larger (30B and 65B) models

[codybum]

I understand that typically llm are undertrained and simply having a larger model does not mean that results will be superior to smaller models. This being said, has there been an attempt to train or fine-tune 30B or 65B LLaMA-based models? If not, what is preventing the effort other than access to physical hardware?

Cody

[haotian-liu]

Hi Cody, thank you for your interest in our work. Currently the main reason is the compute. But we are planning to scale it up if the resources are met. Besides, we are also seeking for other efficient tuning / optimization methods like bitsandbytes.