llama : Homebrew formula

[ggerganov]

Overview

Recently, the llama.cpp project has been added to the official Homebrew Core package manager. This streamlines the installation of the llama.cpp examples and brings convenience to the ecosystem. This discussion is about giving some more visibility to this functionality, highlighting some of the limitations and brainstorming ideas for improving it

Implementation

The llama.cpp Homebrew formula was kindly contributed by members of the @huggingface team:

Homebrew/homebrew-core#172915

Usage

• Install the llama.cpp package:

  brew install llama.cpp

• Run sample completion:

  llama-cli --hf-repo ggml-org/tiny-llamas --hf-file stories15M-q4_0.gguf -n 400 -p "Once opon a time"

• Start an HTTP server:

  llama-server --hf-repo microsoft/Phi-3-mini-4k-instruct-gguf --hf-file Phi-3-mini-4k-instruct-q4.gguf

The commands above take advantage of some of the QoL improvements related to better integration with the HF platform, such as being able to specify HF model ids, download the model automatically using curl and store it in a local cache. For a comprehensive list of these contributions, checkout the llama : Hugging Face integration project (the list is not exhaustive, collaborators are welcome to improve it)

Purpose & Limitations

Having llama.cpp as a Homebrew package can be convenient in some situations because it simplifies the process of obtaining the latest source code and building it. However, keep in mind that the examples in the project (and respectively the binaries provided by the package) are not yet full-blown applications and mostly serve the purpose of demonstrating the functionality of the llama.cpp library. Also, in some environments the installed binaries might not be built with the optimal compile options which can lead to poor performance.

Therefore the recommended way for using llama.cpp remains building it manually from source. Hopefully with time the package and the examples will keep improving and become actual useful tools that can be used in production environments

Discussion

Collaborators are welcome to update this post with up-to-date information and extend it with more useful information and examples. In the comments we can discuss how to improve the contents of the package and improve it's usefulness

[goswamig]

BTW the command to run the sample is

llama-cli --hf-repo ggml-org/tiny-llamas -m stories15M-q4_0.gguf -n 400 -p "Once opon a time"

[rickb-lb]

Hi, I really appreciate all the hard work being done here, it is a great place to learn, thank you!

when I run the above sample command, I get the following output:
Once opon a timeYesadsbutobbutedonobon playerered runatedBadoonlickunersth sett not not not co co co co co co in in in in in in in co co co co co co co co co co in in in in in in in in in one beginning onesOne cole cosh each turnsaster coOneOne co co onele co in beginning in beginningOne co co co co warOnce co co one chamberasteroth course eachzy each one chamberOne beginningzyzyoopzy beginningoboop oneoboopaster whichzy self every beginningothoth course

Not sure what I am doing wrong, but felt it worth reporting

[robbiemu]

how do you run the convert script?

convert_hf_to_gguf.py --help
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/llama.cpp/3889/bin/convert_hf_to_gguf.py", line 29, in <module>
    import gguf
ModuleNotFoundError: No module named 'gguf'

The requirements are not anywhere in the directory I think.