What's the difference between ollama and this?

[Daniel-dev22]

New to this genai world.

What's the difference between running ollama and running this project in docker with Intel iGPU? Besides that this project supports Intel igpu but not ollama.

It appears the API is different between ollama and this project?

https://hub.docker.com/r/ollama/ollama

[marcingomulkiewicz]

Ollama uses llama.cpp under the hood. Both llama-server and ollama support OpenAI API, both do so well enough for typical usecases, but last time I checked, llama-server didn't support tool use (ollama does), plus managing (downloading, switching between, etc.) models is more convenient in ollama. Plus (that's just my opiono) setting various parameters is more straightforward in llama-server than in ollama. YMMV.

[olegshulyakov]

@marcingomulkiewicz What about speed and memory usage? Is there any difference?

[marcingomulkiewicz]

Certainly I haven't noticed any significant differences, I didn't do any precise benchmarks though. Given the fact that llama.cpp is underneath both, I don't expect any noticeable speed differences, and any potential memory overhead is dwarfed by the model itself anyway. I'm unsure how/if things have changed with ollama 0.4.x - it supports image+text->text llamas, so something must have been changed under the hood.