v0.0.21

Dify

Handle: dify
URL: http://localhost:33961/

Self-hosting · Documentation

Dify is an open-source LLM app development platform. Dify's intuitive interface combines AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production.

Starting

# [Optional] Pull the dify images
# ahead of starting the service
harbor pull dify

# Start the service
harbor up dify

harbor open dify

Misc

• Fixed running SearXNG without Webui
• harbor how prompt tuning
• vllm version can now be configured via CLI

v0.0.20

harbor how

$ harbor how to ping a service from another service?
$ harbor how to filter webui error logs with grep?
$ harbor how to make a sandwich?
$ harbor how to run a command in the ollama container?

Harbor can now answer questions about itself! Courtesy of a new cmdh service that is more generally can aid you in shell command generation:

$ harbor cmdh find all images in this folder
$ harbor cmdh stop all running docker containers
$ harbor cmdh download node.js

Note

This is a half-feature and half-gimmick. It can be surprisingly powerful when running with a larger or more grounded model (Codestral, Command-R+)

# See available models
harbor ollama list

# Set a more appropriate model for cmdh
harbor cmdh model codestral

• cmdh service docs

Full Changelog: v0.0.19...v0.0.20

v0.0.19

harbor tunnel - expose your harbor stack over the internet

# I'm feeling adventurous today!
# Expose default UI over the internet
$ harbor tunnel

# We provide plenty of aliases to ensure
# you can pause tunneling even when adventure goes wrong
$ harbor tunnel down
$ harbor tunnel stop
$ harbor tunnel s
$ harbor tunnel d
$ harbor t d
$ harbor t s

# It's a bad idea to expose services without auth to the internet
# But you only live once!
harbor tunnel ollama

The command will print the URL and a QR code to visit.

If you're feeling especially adventurous, you can instruct Harbor to automatically spin up tunnels for specific services whenever it starts:

# Show current services with automatic tunnels
$ harbor tunnels

# See our new config help!
$ harbor tunnels --help
Harbor config: services.tunnels

This field is an array, use the following actions to manage it:

  ls            - List all values
  clear         - Remove all values
  rm <value>    - Remove a value
  rm <index>    - Remove a value by index
  add <value>   - Add a value

# Add a service
$ harbor tunnels add webui

# Harbor will start the tunnel automatically now
$ harbor up

Current version is an initial implementation with a free version of cloudflared. If you know how to use Docker and Compose you can already tweak Harbor to use your CloudFlare account, otherwise - stay tuned for more updates.

Misc

• harbor ls [-a] list all available or currently active services
• harbor cfd access to containerized cloudflared with the access to the rest of harbor services
• config aliases got help entries (both string and array counterparts)
• harbor url now supports muiltiple kinds of URLs:
  • -i, --intra - URL of the service on the harbor's docker network
  • -a, --adressable, --lan - (supposed) URL of the service in your LAN (used by harbor qr)
  • no arguments - URL on your local host

---

Full Changelog: v0.0.18...v0.0.19

v0.0.18

harbor qr <handle>

Poke at your fancy new LLM setup right from your phone.

# This service will open by default
$ harbor config get ui.main

# Generate a QR code for default UI
$ harbor qr

# Generate a QR code for a specific service
# Makes little sense for non-UI services.
$ harbor qr ollama

Example

Full Changelog: v0.0.17...v0.0.18

v0.0.17

Open Interpreter integration

Open Interpreter lets LLMs run code (Python, Javascript, Shell, and more) locally. You can chat with Open Interpreter through a ChatGPT-like interface in your terminal.

# Pre-build the image for convenience
harbor build opint

# interpreter likes juicy large LLMs
harbor opint model codestral

cd ~/my-project

# Work together with interpreter
harbor opint

• Integrated out of the box with all OpenAI-compatible backends in Harbor
• Service Wiki
• harbor opint CLI docs

Full Changelog: v0.0.16...v0.0.17

v0.0.16

mistral.rs integration

Handle: mistralrs
URL: http://localhost:33951

Mistral.rs is a fast LLM inference platform supporting inference on a variety of devices, quantization, and easy-to-use application with an Open-AI API compatible HTTP server and Python bindings.

Defaults

# Spin up harbor with mistral.rs service
harbor up mistralrs

# Launch the default UI, pre-configured with mistral.rs
harbor open

Plain Models

# For "plain" models:
# Download the model to the global HF cache
harbor hf download IlyaGusev/gemma-2-2b-it-abliterated

# Set model/type/arch
harbor mistralrs model IlyaGusev/gemma-2-2b-it-abliterated
harbor mistralrs type plain
harbor mistralrs arch gemma2

# Gemma 2 doesnt't support paged attention
harbor mistralrs args --no-paged-attn

# Run with the default settings
harbor up mistralrs

GGUF Models

# Set the model type to GGUF
harbor mistralrs type gguf

# - Unset ISQ off, as it's not supported
# for GGUF models
# - For GGUFs, architecture is inferred from the file
harbor mistralrs isq ""
harbor mistralrs arch ""

# Use "folder" specifier to point to the model
# "hf/full/path"  - mounted HF cache. Note that you need
#                   a full path to the folder with .gguf
# "-f Model.gguf" - the model file
harbor mistralrs model "hf/hub/models--microsoft--Phi-3-mini-4k-instruct-gguf/snapshots/999f761fe19e26cf1a339a5ec5f9f201301cbb83/ -f Phi-3-mini-4k-instruct-q4.gguf"

# When configured, launch
harbor up mistralrs

CLI

# Show service API docs (local, when running)
harbor mistralrs docs
# Check if the service is health
harbor mistralrs health
# Call original mistralrs-server CLI
harbor mistralrs --help
# Debug the installation
harbor shell mistralrs

Better linking and aliases

harbor link is now configurable - choose link names and location.

# Assuming it's not linked yet

# See the defaults
./harbor.sh config get cli.path
./harbor.sh config get cli.name
./harbor.sh config get cli.short

# Customize
./harbor.sh config set cli.path ~/bin
./harbor.sh config set cli.name ai
./harbor.sh config set cli.short ai

# Link
./harbor.sh ln
# Include the short link
./harbor.sh ln --short
# Unlink
harbor unlink

Misc

• .nvidia. files are now implemented with cross-service file behavior
• multiple new aliases, see the Wiki and harbor -h for reference

Full Changelog: v0.0.15...v0.0.16

v0.0.15

Plandex integration

# Run plandex service
harbor up plandex
# Run healthcheck against the Plandex service
harbor plandex health # should print "OK"

• Service overview
• Setup/configuration with Ollama guide in the Wiki
• harbor pdx CLI reference

Local env

.env file can now be used for any necessary permanent overrides without risking to conflict with updates from upstream.

To accompany the change:

# Brings the configuration back to current defaults
harbor config reset

Misc

• HuggingFace CLI is a service - unify config management with other services
• Poor man's harbor update

---

Full Changelog: v0.0.14...v0.0.15

v0.0.14

PAR LLAMA

TUI for Ollama.

PAR LLAMA is an amazing terminal UI for interacting with Ollama. Everything you'd expect from a modern chat interface, but in the terminal.

# Ollama should be one of the
# running services in order to be reachable by parllama
harbor up ollama

# 1. Shortcut
harbor parllama

# 2. Via harbor run - the underlying command is different
harbor run parllama

# 3. Via interactive service shell
harbor shell parllama
$ parllama

See service wiki for more details

Full Changelog: v0.0.13...v0.0.14

v0.0.13

What's Changed

TabbyAPI backend support

Wiki Docs

harbor tabbyapi model Annuvin/gemma-2-2b-it-abliterated-4.0bpw-exl2
harbor up tabbyapi

New CLI Features

harbor hf dl

Integrating awesome HuggingFaceModelDownloader CLI for easier HF/Llama.cpp cache management

# See the original help
harbor hf dl --help

# EXL2 example
#
# -s ./hf - Save the model to global HuggingFace cache (mounted to ./hf)
# -c 10   - make download go brr with 10 concurrent connections
# -m      - model specifier in user/repo format
# -b      - model revision/branch specifier (where applicable)
harbor hf dl -c 10 -m turboderp/TinyLlama-1B-exl2 -b 2.3bpw -s ./hf

# GGUF example
#
# -s ./llama.cpp - Save the model to global llama.cpp cache (mounted to ./llama.cpp)
# -c 10          - make download go brr with 10 concurrent connections
# -m             - model specifier in user/repo format
# :Q2_K          - file filter postfix - will only download files with this postfix
harbor hf dl -c 10 -m TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF:Q2_K -s ./llama.cpp

harbor hf find

To accompany the hf dl - a quick way to jump right to the HF Hub to find new models.

harbor hf find gguf
harbor hf find exl2 gemma-2
harbor hf find awq llama-3.1
harbor hf find tinyllama

Misc

• docs: update README.md by @eltociear in #3
• harbor shell - launch interactive shell in service container (shortcut from previous harbor exec + harbor cmd combinations)
• harbor build - for services that'll have their Dockerfile within Harbor repo (such as hfdownloader)

New Contributors

• @eltociear made their first contribution in #3

Full Changelog: v0.0.12...v0.0.13

v0.0.12

• Container name prefix: all containers from the toolkit are now prefixed with harbor. to avoid conflict with other locally running setups

# Can be adjusted via .env or with CLI
harbor config get container.prefix # harbor
harbor config set container.prefix friendly # friendly.ollama

• Fixing harbor ln to use CLI name without an extension

Full Changelog: v0.0.11...v0.0.12