v0.1.27 - Harbor Boost

v0.1.27 - Harbor Boost

RCN Llama 3.1 8B + Web RAG in Open WebUI

Harbor can now boost small llamas to be better at creative and reasoning tasks. I'm happy to present Harbor Boost - optimizing LLM proxy with OpenAI-compatible API.

It allows implementing workflows like below:

• When "random" is mentioned in the message, klmbr will rewrite 35% of message characters to increase the entropy and produce more diverse completion
• Launch self-reflection reasoning chain when the message ends with a question mark
• Expand the conversation context with the "inner monologue" of the model, where it can iterate over your question a few times before giving the final answer
• Count "r"s in "strawberry" this problem is solved

See how Harbor can boost the creativity _randomness in a small llama beyound the infinite "Turquoise", using klmbr:

Screencast.from.22-09-24.17.41.52.webm

klmbr will process your inputs to inject some randomness into them, so even with 0 temperature - LLM output will be varied (sometimes in a very unexpected way). Harbor allows to configure various parameters of klmbr via both CLI and .env.

You can also use rcn (brand new technique) an g1 CoT to make your llama more reasonable.

This works, essentially, by just giving an LLM more time to "think" about its answer and improves reasoning in many cases at the expense of larger amount of tokens consumed.

• Harbor Boost docs

Misc

• harbor size - shows the size of caches from Harbor services on your system (we don't recomment running it, it hurts)
• harbor bench - better logs with ETA and service pointers, fixed issue with parameter propagation for reproducible results, added BBH256/32 examples
• harbor update should now allow updating past 0.1.9 on MacOS (granted you'll manage to update past it in the first place 🙃)

Full Changelog: v0.1.26...v0.1.27