feat: 2.5x faster model generation (and 2.5x lower model costs)

[moinnadeem]

Hello!

🔥 work on the companion app!

I've been hacking around on low-latency inference recently on Replicate. This PR swaps out the default Vicuna model for a model with ~2.5x faster generation (depending on the sequence length).

It's just a base Replicate model, so we can just swap out the model string: https://replicate.com/moinnadeem/fastervicuna_13b
We could also work on getting these things upstreamed to the main Replicate model too.

WIP:

• Test the frontend
• a 3x improvement on latency

Things to test:

• Does streaming generation work? I am in progress on this one

[jenniferli23]

Thanks for the PR @moinnadeem !!

Do you mind elaborate on how did you achieve the 2.5 faster and 2.5 lower cost model? And how did you benchmark it?

We're working on testing this out before merging.