Is there high-level API for LLama.cpp on C++?

[AymenSekhri]

I am trying to integrate LLama.cpp in a C++ project that I have and I am looking for high level APIs to interact with it, the closest thing that I got was in examples/main, however it contains a lot of low level details, and it outputs a lot of debug and hardware information and that I can't find a way to disable.

Python have these nice high level API:

from llama_cpp import Llama

llm = Llama(
      model_path="./models/7B/llama-model.gguf",
      # n_gpu_layers=-1, # Uncomment to use GPU acceleration
      # seed=1337, # Uncomment to set a specific seed
      # n_ctx=2048, # Uncomment to increase the context window
)
output = llm(
      "Q: Name the planets in the solar system? A: ", # Prompt
      max_tokens=32, # Generate up to 32 tokens, set to None to generate up to the end of the context window
      stop=["Q:", "\n"], # Stop generating just before the model would generate a new question
      echo=True # Echo the prompt back in the output
) # Generate a completion, can also call create_completion
print(output)

Rust has one too:

use llama_cpp::{LlamaModel, LlamaParams, SessionParams};
use llama_cpp::standard_sampler::StandardSampler;

// Create a model from anything that implements `AsRef<Path>`:
let model = LlamaModel::load_from_file("path_to_model.gguf", LlamaParams::default()).expect("Could not load model");

// A `LlamaModel` holds the weights shared across many _sessions_; while your model may be
// several gigabytes large, a session is typically a few dozen to a hundred megabytes!
let mut ctx = model.create_session(SessionParams::default()).expect("Failed to create session");

// You can feed anything that implements `AsRef<[u8]>` into the model's context.
ctx.advance_context("This is the story of a man named Stanley.").unwrap();

// LLMs are typically used to predict the next word in a sequence. Let's generate some tokens!
let max_tokens = 1024;
let mut decoded_tokens = 0;

// `ctx.start_completing_with` creates a worker thread that generates tokens. When the completion
// handle is dropped, tokens stop generating!

let mut completions = ctx.start_completing_with(StandardSampler::default(), 1024).into_strings();

for completion in completions {
    print!("{completion}");
    let _ = io::stdout().flush();

    decoded_tokens += 1;

    if decoded_tokens > max_tokens {
        break;
    }
}

Is there one for C++ as well that I missed?

[ngxson]

We actually planned to add one since a long time ago: #5215

[ggerganov]

It's still on the roadmap, we will implement it eventually

[AymenSekhri]

It's still on the roadmap, we will implement it eventually

Is that roadmap documented somewhere?, I might able to implement it.

[ngxson]

@AymenSekhri We haven't yet had an agreement on what to do for this task (no documentation).

If you have an idea, you can create a PR or leave a comment on #5215

[RhinoDevel]

In the meantime, looking at the simple example also helps (helped me, at least).

[HuangLED]

why do you integrate, instead of running service independently then call the service from you application?

trying to learn from other folks' best practice

[aguzev]

1. Another cog in the machine adds friction, e.g. forces to learn and maintain another integration technology. Often it adds dependencies on integration layer libraries. This kind of choice must be justified.
2. Resources. Little to no data movement. Not mentioning serialization.

[stellarpower]

I'm guessing C-like C++ is kinda necessary for bindings to other languges. I'm only just taking a first look through the source code, but if the details are somewhat low-level and based on some of the C-like style I'm seeing I'd assume that's part of the reason. But I'm also interested because C++ should in general be more optimisable than C, especially with some of the modern features since say C++14. Personally, I generally write in modular C++ and then try to wrap and bind in for C if I need to than the other way round. Out of interest, has this ever been considered, both from performance and maintainability?

[aguzev]

I found nice C++ code under the hood of llama.cpp pure C interface. It provides RAII, for example. The problem is using it directly from a C++ program. I even asked another question #9880.

I tried to edit the llama.cpp source to expose pure C++ implementation and found it too expensive. It requires refactoring and breaks updates from actively developed original repo. Now I'm wrapping C functions with C++ classes. Quite a waste of time.

[stellarpower]

Fair enough, I'm still yet to dive in much. But only having played with for a day, I agree, I'm doing the same, which is a shame. RAII in general is a must in my view, it wouldn't be too difficult to keep the C functions but add overloads for unique_ptr with the right deleters added in. With some effort, in the past I have also used #ifdef __cplusplus to make a struct and only have a C++ compiler see the important bits, then extern C some trampolines for the free function equivalents. That worked well for RAII, I even had some TMP in there. So yeah, there are some ways to make C++ first-class and C to wrap it without copy-paste boilerplate, it can be a little unpleasant to look at and require some tricks, but it's doable, whilst lots of projects just do the reverse.

To the maintainers - being honest, I think refactoring what is there - not just the headers, but especially the examples - would make sense logical before trying to come up with and fix on a particular API. And in fact, one might reveal itself more naturally. 600-line functions with everything in main aren't really a great thing to have. Personally when I'm approaching a codebase I usually learn how it works by doing that, breaking it up, wrapping C in C++, getting rid of the repetition, replacing lots of if statements with exceptions, etc. It makes it easier to maintain and understand in the longer term.

Probably stating the obvious here, but it'd likely also make sense to have some parallels between the package for other languages, help keep the overlap large, so maybe some feedback form users there or using what works well and does not in those might give some inspiration to the format. But of course, it wouldn't be much good having pythonic C++ or anything!

[AymenSekhri]

@stellarpower I fully agree...
After I posted this issue, I tried to make a C++ wrapper after many attempts to refactor the examples and trying to figure out the necessary configurations to be passed. I had a working draft but now it is broken after few updates that changed the interfaces entirely.
I don't think it is a good idea to keep such a project in pure C, specially if there are plenty of functions with 600 LOC and many ambiguous function names (this might be because I am not very familiar with LLMs architecture, but there is plenty of room to improve here)

Binding with other languages doesn't necessary require sticking the entire project to pure C, the project could be in modern C++ while having C interface that is wrapping around a C++ higher level interfaces.

[stellarpower]

Don't suppose, are either of these wrappers/other code you've written in this endeavour available online in any way? I have the barebones but it'd be nice if there were a simple example included in the repository of a bidirectional chat, other than trying to pull it out of the very lengthy main CLI application or the server. I have this gist that I'll begin pulling apart, which is at least smaller, but haven't run it yet and it suffers the same issue really, just less of it.