Add Orca if it ever releases

[timjzee]

Whitepaper: https://arxiv.org/pdf/2306.02707.pdf

Will be released here: https://aka.ms/orca-lm

Summary: https://www.youtube.com/watch?v=Dt_UNg7Mchg

[mdingemanse]

Interesting find. And more evidence of the growing importance of synthetic instruction tuning data.

[mdingemanse]

Looks like there may be some version of it here: https://huggingface.co/yhyhy3/med-orca-instruct-33b-GPTQ

[timjzee]

I think most of the "Orca" models on Huggingface are projects which used a similar approach to the one described in the Microsoft paper. AFAIK they are not actual Orca releases.

[mdingemanse]

Nah. There is a new preprint that says

We open-source Orca 2 to encourage further research on the development, evaluation, and alignment of smaller LMs.

But nothing is open-sourced; this is a Llama2 finetune where only the instruction-tuned (or what they call explanation-tuned) model weights are made available, but none of instruction/explanation datasets and none of the source code is made available.

Thanks Meta for thoroughly diluting the term open source and thanks Microsoft for further contributing to it.