Can't access llama-server from outside although I have exposed ports.

[opendeluxe]

I started llama-server on a Hetzner server with the following command:

docker run -p 8000:8000 --gpus all -v ./models:/models ghcr.io/ggerganov/llama.cpp:server-cuda -m ...  --n-gpu-layers 99 --ctx_size 1024 --host 0.0.0.0 --port 8000 

In the terminal of the host system, I can curl it: curl localhost:8000
It prints html and javascript to the console.

Shouldn't I be also able to access this application from outside my server via web browser? I have exposed the ports via docker using the switch "-p 8000:8000".

When I make a request to the IP-adress of the server and the corresponding port 8000, it just hangs, eventually timing out later.
I even can not see any new log entry of llama-server in that time.

I don't run any firewall. ufw status shows inactive.

[itinance]

Install nginx as reverse proxy. Here is a (simple) example config:

server {
    listen 80;
    server_name domain.com;
location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

[dspasyuk]

curl localhost:8000

@opendeluxe what happen when you try to curl host_system_ip:8000 from the host system? Does it freeze too?

[opendeluxe]

curl localhost:8000

@opendeluxe what happen when you try to curl host_system_ip:8000 from the host system? Does it freeze too?

locally on the host it works perfectly

[dspasyuk]

hmm, could you try:
sudo ufw enable
sudo ufw allow 8000

[99991]

I'd recommend against exposing llama-server publicly for security reasons. Instead, forward the remote port to your local machine using ssh:

ssh -L 8000:localhost:8000 yourusername@yourserver

Then you should be able to access http://localhost:8000 in your browser.

[ngxson]

The problem you're mentioning is not related to llama.cpp. Most VPS provider block all ports by defualt, you need to allow some of them manually.

For hetzner: https://docs.hetzner.com/konsoleh/account-management/configuration/openports/