This project provides a Node.js server for a chat user interface (UI) that interacts with the Llama.cpp library. It allows users to communicate with the llama.cpp application via a web-based chat interface.
Starting from version 0.24 model will be downloaded automatically.
git clone https://github.com/ggerganov/llama.cpp.git; cd llama.cpp; sed -i 's/-arch=native/-arch=all/g' Makefile; make clean && LLAMA_CUDA=1 make -j 6; cd ..; git clone https://github.com/dspasyuk/llama.cui; cd llama.cui; npm install; node server.js
git clone https://github.com/ggerganov/llama.cpp.git; cd llama.cpp; sed -i 's/-arch=native/-arch=all/g' Makefile; make clean && make -j 6; cd ..; git clone https://github.com/dspasyuk/llama.cui; cd llama.cui; npm install; node server.js
git clone https://github.com/ggerganov/llama.cpp.git; cd llama.cpp; sed -i 's/-arch=native/-arch=all/g' Makefile; make clean && make -j 6; cd ..; git clone https://github.com/dspasyuk/llama.cui; cd llama.cui; npm install; node server.js
Change "--n-gpu-layers" in config.js file depending on the type of architecture used and available VRAM. For the default model (Llama3-instruct) this should be equal to 35, for compatibility it is currently set to 25, you will need at least 6Gb of VRAM to run the model, so Nvidia GTX1060 and above is a must.
-
Clone the repository:
git clone https://github.com/ggerganov/llama.cpp.git -
Build Lllama.cpp with GPU or CPU support
cd llama.cppsed -i 's/-arch=native/-arch=all/g' Makefile# could be skipped if native arch worksmake clean && LLAMA_CUDA=1 make -j 4# for GPU CUDA versionmake clean && LLAMA_CUBLAS=1 make -j 4# for GPU cuBLAS versionor
make# for CPU version -
Clone llama.cui
git clone https://github.com/dspasyuk/llama.cui -
Download LLM model from hugging face in GGUF format, for example:
a. Meta-Llama-3-8B-Instruct: https://huggingface.co/QuantFactory/Meta-Llama-3-8B-Instruct-GGUF
or https://huggingface.co/dspasyuk/Meta-Llama-3-8B-Instruct-Q5_K_S-GGUF/blob/main/Meta-Llama-3-8B-Instruct-Q5_K_S.ggufb. Dolphin-Mistral 7B: https://huggingface.co/TheBloke/dolphin-2.1-mistral-7B-GGUF/blob/main/dolphin-2.1-mistral-7b.Q5_0.gguf
c. Einstein-v4-7B: https://huggingface.co/LoneStriker/Einstein-v4-7B-GGUF
d. Qwen2-7B-Instruct Models: https://huggingface.co/Qwen/Qwen2-7B-Instruct-GGUF/tree/main (will need flash attentioon enabled in config.js e.g. -fa) (Default)
-
Install the project and set your configuration parameters
`cd llama.cui
npm install`
Open config.js and change the hostname, port, path to llama.cpp main file, and the model name/path
To run just type:
npm start
Default login and password are specified in the config file but could be easily integrated with the user database. The login is currently set to false. To enable login set login to true in the config file and change password.
As of version 1.15 the llama.cui supports Piper for a text-to-voice generation. Enable it in config.js, make sure to install Piper before running llama.cui
git clone https://github.com/rhasspy/piper.git
cd piper
make
That should build Piper and put it in "piper/install/"
Models can be found on Hugging Face:
https://huggingface.co/rhasspy/piper-voices
Default Llama.cui voice model is librits/en_US-libritts_r-medium.onnx"
https://huggingface.co/rhasspy/piper-voices/tree/main/en/en_US/libritts/high
// The config below is already a default in config.js. You will only need to set "enabled" to true in piper config
config.piper = {
enabled: true,
rate: 20500, // depends on your model
output_file: 'S16_LE', //Piper outputs 16-bit mono PCM buffers so keep this value as is
exec: "../../piper/install/piper", // set a path to your piper installation
model: "/home/denis/CODE/piper/models/librits/en_US-libritts_r-medium.onnx" // set a path to your voice models
};
In addition to the regular Linux instructions on Mac other configurations must be performed to install Piper.
`bash piper_install_mac.sh`
echo 'Welcome to the world of speech synthesis!' | "$PIPER_ROOT_FOLDER/piper/install/piper" \ --model "$PIPER_ROOT_FOLDER/models/librits/en_US-libritts-high.onnx" \ --output-file welcome.wav
If it fails at any stage try the guide below:
First lets install brew if you do not have it yet:
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
/opt/homebrew/bin/brew install espeak-ng
You now should have /opt/homebrew/Cellar/espeak-ng/1.51/lib/libespeak-ng.1.dylib //your version might be different
cd to your piper root folder
git clone https://github.com/rhasspy/piper-phonemize.git
cd piper-phonemize
make
Once the compilation process is done you should have libpiper_phonemize.1.dylib in ./piper-phonemize/install/lib
Now lets create the necessary links to the libraries so that piper can find them:
Add this lines to your ~/.zprofile file before 'export PATH':
`PATH="/opt/homebrew/bin:${PATH}"
export DYLD_LIBRARY_PATH=/opt/homebrew/Cellar/espeak/1.48.04_1/lib/:$DYLD_LIBRARY_PATH
export DYLD_LIBRARY_PAT=/PIPER_ROOT_DEER/piper-phonemize/lib:$DYLD_LIBRARY_PATH`
make sure you provide the correct path to PIPER_ROOT_DEER
Do not forget to source your env file:
source ~/.zprofile
You should now be able to run piper as following, make sure the path to your piper install is correct: echo 'Welcome to the world of speech synthesis!' | \ ./piper/install/piper --model ./piper/models/librits/en_US-libritts-high.onnx --output-file welcome.wav
piper_llcui.mp4
llama.cui supports embeddings from a text file (see the docs folder), MongoDB, and Web (duckduckgo).
You will need to delete the existing DB folder before running llama.cui. The new database will be generated on the next request for embedding (select use database in the bottom left corner of the UI interface to generate the database)
For data format convention, llama.cui uses the anytotext.js library. You can place any doc, xlsx, docx, txt, or other text files into "docs" directory to create your vector database. All embeddings are treated localy using all-MiniLM-L6-v2 model.
