GPU Tabby: "Error response from daemon: driver failed programming external connectivity on endpoint" #2265

Geremia · 2024-05-24T19:55:30Z

Geremia
May 24, 2024

Describe the bug

$ docker run -it --gpus all   -p 8080:8080 -v $HOME/.tabby:/data   tabbyml/tabby serve --model StarCoder-1B --device cuda
docker: Error response from daemon: driver failed programming external connectivity on endpoint boring_nightingale (5290e0be6de9317e87af6dcf9b3ee98772c3bf1f7a6914a69855f074b90473d5): Error starting userland proxy:.
ERRO[0000] error waiting for container:

Information about your version
Please provide output of tabby --version

tabby 0.11.1

Information about your GPU

`nvidia-smi`

Fri May 24 12:53:27 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Quadro RTX 4000                Off |   00000000:42:00.0  On |                  N/A |
| 30%   42C    P8             16W /  125W |    1514MiB /   8192MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      3863      G   /usr/libexec/Xorg                             749MiB |
|    0   N/A  N/A      3966      G   /usr/bin/kwin_x11                              16MiB |
|    0   N/A  N/A      3989      G   /usr/bin/plasmashell                           44MiB |
|    0   N/A  N/A      4249      G   /usr/bin/kalendarac                             3MiB |
|    0   N/A  N/A      4321      G   /usr/lib64/thunderbird/thunderbird            264MiB |
|    0   N/A  N/A      4384      G   /usr/bin/akonadi_archivemail_agent              3MiB |
|    0   N/A  N/A      4390      G   .../bin/akonadi_followupreminder_agent          3MiB |
|    0   N/A  N/A      4408      G   /usr/bin/akonadi_mailfilter_agent               3MiB |
|    0   N/A  N/A      4412      G   /usr/bin/akonadi_mailmerge_agent                3MiB |
|    0   N/A  N/A      4426      G   /usr/bin/akonadi_sendlater_agent                3MiB |
|    0   N/A  N/A      4429      G   /usr/bin/akonadi_unifiedmailbox_agent          75MiB |
|    0   N/A  N/A      4660      G   /usr/lib64/firefox/firefox                    179MiB |
|    0   N/A  N/A     19761      G   /opt/calibre/bin/calibre                       75MiB |
|    0   N/A  N/A     47826      G   /usr/bin/krunner                                6MiB |
|    0   N/A  N/A     48258      G   /usr/bin/python3                               75MiB |
+-----------------------------------------------------------------------------------------+

wsxiaoys · 2024-05-27T23:04:02Z

wsxiaoys
May 27, 2024
Maintainer

Can you provide the full output / screenshot for the command?

docker run -it --gpus all \
  -p 8080:8080 -v $HOME/.tabby:/data \
  tabbyml/tabby serve --model StarCoder-1B --device cuda

0 replies

Geremia · 2024-05-27T23:05:57Z

Geremia
May 27, 2024
Author

@wsxiaoys

Can you provide the full output / screenshot for the command?

$ docker run -it --gpus all \
>   -p 8080:8080 -v $HOME/.tabby:/data \
>   tabbyml/tabby serve --model StarCoder-1B --device cuda
docker: Error response from daemon: driver failed programming external connectivity on endpoint dreamy_colden (489cb82953a3308a63236d72837235db68dea933a4b1ad8ee49ff0719a7cb482): Error starting userland proxy:.
ERRO[0000] error waiting for container:

docker container inspect 8b899767d90a

[
    {
        "Id": "8b899767d90a65292d524b9bbd22eaa7f48278db6904ef21b980cc76da893161",
        "Created": "2024-05-27T23:08:15.30876045Z",
        "Path": "/opt/tabby/bin/tabby",
        "Args": [
            "serve",
            "--model",
            "StarCoder-1B",
            "--device",
            "cuda"
        ],
        "State": {
            "Status": "created",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 128,
            "Error": "driver failed programming external connectivity on endpoint peaceful_hawking (b3360cb8df8e0ffbe9eeb61841e10cde99264cb9c21673f738ddec2f08d6df38): Error starting userland proxy: ",
            "StartedAt": "0001-01-01T00:00:00Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:35856381004daa40e39c1b132a8ae5d6b982783ce652007f20ef632283667e7d",
        "ResolvConfPath": "/var/lib/docker/containers/8b899767d90a65292d524b9bbd22eaa7f48278db6904ef21b980cc76da893161/resolv.conf",
        "HostnamePath": "",
        "HostsPath": "/var/lib/docker/containers/8b899767d90a65292d524b9bbd22eaa7f48278db6904ef21b980cc76da893161/hosts",
        "LogPath": "",
        "Name": "/peaceful_hawking",
        "RestartCount": 0,
        "Driver": "btrfs",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/home/geremia/.tabby:/data"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {
                "8080/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "8080"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                84,
                147
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": [],
            "BlkioDeviceWriteBps": [],
            "BlkioDeviceReadIOps": [],
            "BlkioDeviceWriteIOps": [],
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": [
                {
                    "Driver": "",
                    "Count": -1,
                    "DeviceIDs": null,
                    "Capabilities": [
                        [
                            "gpu"
                        ]
                    ],
                    "Options": {}
                }
            ],
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": null,
            "Name": "btrfs"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/home/geremia/.tabby",
                "Destination": "/data",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "8b899767d90a",
            "Domainname": "",
            "User": "",
            "AttachStdin": true,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "8080/tcp": {}
            },
            "Tty": true,
            "OpenStdin": true,
            "StdinOnce": true,
            "Env": [
                "PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/tabby/bin",
                "NVARCH=x86_64",
                "NVIDIA_REQUIRE_CUDA=cuda>=11.7 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471",
                "NV_CUDA_CUDART_VERSION=11.7.99-1",
                "NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-7",
                "CUDA_VERSION=11.7.1",
                "LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64",
                "NVIDIA_VISIBLE_DEVICES=all",
                "NVIDIA_DRIVER_CAPABILITIES=compute,utility",
                "NV_CUDA_LIB_VERSION=11.7.1-1",
                "NV_NVTX_VERSION=11.7.91-1",
                "NV_LIBNPP_VERSION=11.7.4.75-1",
                "NV_LIBNPP_PACKAGE=libnpp-11-7=11.7.4.75-1",
                "NV_LIBCUSPARSE_VERSION=11.7.4.91-1",
                "NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-7",
                "NV_LIBCUBLAS_VERSION=11.10.3.66-1",
                "NV_LIBCUBLAS_PACKAGE=libcublas-11-7=11.10.3.66-1",
                "NV_LIBNCCL_PACKAGE_NAME=libnccl2",
                "NV_LIBNCCL_PACKAGE_VERSION=2.13.4-1",
                "NCCL_VERSION=2.13.4-1",
                "NV_LIBNCCL_PACKAGE=libnccl2=2.13.4-1+cuda11.7",
                "NVIDIA_PRODUCT_NAME=CUDA",
                "TABBY_ROOT=/data"
            ],
            "Cmd": [
                "serve",
                "--model",
                "StarCoder-1B",
                "--device",
                "cuda"
            ],
            "Image": "tabbyml/tabby",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/opt/tabby/bin/tabby"
            ],
            "OnBuild": null,
            "Labels": {
                "maintainer": "NVIDIA CORPORATION <[email protected]>",
                "org.opencontainers.image.created": "2024-05-15T05:00:56.710Z",
                "org.opencontainers.image.description": "Self-hosted AI coding assistant",
                "org.opencontainers.image.licenses": "NOASSERTION",
                "org.opencontainers.image.ref.name": "ubuntu",
                "org.opencontainers.image.revision": "c3eb1c1312277e000096f22421c1014a3287e14a",
                "org.opencontainers.image.source": "https://github.com/TabbyML/tabby",
                "org.opencontainers.image.title": "tabby",
                "org.opencontainers.image.url": "https://github.com/TabbyML/tabby",
                "org.opencontainers.image.version": "0.11.1"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "559e59772325cb4fb83b3ab9f0683580477f245fda0125a1377769c58c89a7f8",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/559e59772325",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "3117c4ec1d090b42d6d02645c8b5297a562cc0c041ac6ccbc53f568bd3bf3519",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

Also, I'm running Slackware, not Ubuntu.
Nothing's already using port 8080, and I tried to change the port, but I still get the same issue.

0 replies

wsxiaoys · 2024-05-27T23:24:42Z

wsxiaoys
May 27, 2024
Maintainer

Have you installed the NVIDIA container Toolkit?

https://tabby.tabbyml.com/docs/quick-start/installation/docker/

0 replies

Geremia · 2024-05-27T23:34:47Z

Geremia
May 27, 2024
Author

@wsxiaoys

Have you installed the NVIDIA container Toolkit?

Yes, but maybe I didn't set it up properly?

I ran (according to the instructions):

nvidia-ctk runtime configure --runtime=docker

and restarted the Docker daemon, but I still get the same docker: Error response from daemon: driver failed programming external connectivity on endpoint… issue.
My /etc/docker/daemon.json is:

{
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }
}

0 replies

Geremia · 2024-05-28T02:01:53Z

Geremia
May 28, 2024
Author

docker run --entrypoint /opt/tabby/bin/tabby-cpu tabbyml/tabby serve works, but the GPU starting command still doesn't.
Is GPU tabby command really not there or something?

0 replies

wsxiaoys · 2024-05-28T03:24:46Z

wsxiaoys
May 28, 2024
Maintainer

It's very likely related to your setup (Slackware / CUDA / Nvidia CUDA Container runtime), as it's not something we've previously adapted. Ubuntu is kind of the only battle-tested platform for CUDA-related tasks (not just Tabby, but the whole ecosystem).

One way to try is to get the binary distribution (instead of the Docker distribution) from https://github.com/TabbyML/tabby/releases/tag/v0.11.1 to see if it works on Slackware, as it doesn't require the CUDA container runtime.

0 replies

Geremia · 2024-05-28T03:42:34Z

Geremia
May 28, 2024
Author

@wsxiaoys
Also, $ docker run --entrypoint /opt/tabby/bin/tabby tabbyml/tabby serve --help
gives:
/opt/tabby/bin/tabby: error while loading shared libraries: libcuda.so.1: cannot open shared object file: No such file or directory

0 replies

Geremia · 2024-05-28T03:57:08Z

Geremia
May 28, 2024
Author

@wsxiaoys It seems to be a network issue.

$ docker run --entrypoint /opt/tabby/bin/tabby-cpu tabbyml/tabby serve 

  TELEMETRY

  As an open source project, we collect usage statistics to inform development priorities. For more
  information, read https://tabby.tabbyml.com/docs/configuration#usage-collection
  
  We will not see or any code in your development process.

  Welcome to Tabby!

  If you have any questions or would like to engage with the Tabby team, please join us on Slack
  (https://links.tabbyml.com/join-slack-terminal).



████████╗ █████╗ ██████╗ ██████╗ ██╗   ██╗
╚══██╔══╝██╔══██╗██╔══██╗██╔══██╗╚██╗ ██╔╝
   ██║   ███████║██████╔╝██████╔╝ ╚████╔╝ 
   ██║   ██╔══██║██╔══██╗██╔══██╗  ╚██╔╝  
   ██║   ██║  ██║██████╔╝██████╔╝   ██║   
   ╚═╝   ╚═╝  ╚═╝╚═════╝ ╚═════╝    ╚═╝   

📄 Version 0.11.1
🚀 Listening at 0.0.0.0:8080

Yet, lsof -i :8080 returns no PIDs listening on port 8080.

0 replies

wsxiaoys · 2024-05-28T04:07:41Z

wsxiaoys
May 28, 2024
Maintainer

Yet, lsof -i :8080 returns no PIDs listening on port 8080.

Need append something like -p 8080:8080 to expose container port

0 replies

Geremia · 2024-05-28T04:08:05Z

Geremia
May 28, 2024
Author

Specifying -p 8080:8080 is what tripped it up.

docker run -ti --gpus all -v $HOME/.tabby:/data tabbyml/tabby serve --model StarCoder-1B --device cuda worked, downloaded the StarCoder-1B model, but it's not listening:

$ docker run -ti --gpus all -v $HOME/.tabby:/data   tabbyml/tabby serve --model StarCoder-1B --device cudaWriting to new file.
🎯 Downloaded https://huggingface.co/TabbyML/models/resolve/main/starcoderbase-1B.Q8_0.gguf to /data/models/TabbyML/StarCoder-1B/ggml/model.gguf.tmp
   00:01:31 ▕████████████████████▏ 1.23 GiB/1.23 GiB  13.78 MiB/s  ETA 0s.                                                                         ggml_cuda_init: GGML_CUDA_FORCE_MMQ:   no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 1 CUDA devices:
  Device 0: Quadro RTX 4000, compute capability 7.5, VMM: yes

  TELEMETRY

  As an open source project, we collect usage statistics to inform development priorities. For more
  information, read https://tabby.tabbyml.com/docs/configuration#usage-collection
  
  We will not see or any code in your development process.

  Welcome to Tabby!

  If you have any questions or would like to engage with the Tabby team, please join us on Slack
  (https://links.tabbyml.com/join-slack-terminal).



████████╗ █████╗ ██████╗ ██████╗ ██╗   ██╗
╚══██╔══╝██╔══██╗██╔══██╗██╔══██╗╚██╗ ██╔╝
   ██║   ███████║██████╔╝██████╔╝ ╚████╔╝ 
   ██║   ██╔══██║██╔══██╗██╔══██╗  ╚██╔╝  
   ██║   ██║  ██║██████╔╝██████╔╝   ██║   
   ╚═╝   ╚═╝  ╚═╝╚═════╝ ╚═════╝    ╚═╝   

📄 Version 0.11.1
🚀 Listening at 0.0.0.0:8080

0 replies

Geremia · 2024-05-28T04:08:31Z

Geremia
May 28, 2024
Author

@wsxiaoys

Need append something like -p 8080:8080 to expose container port

That's what triggers ERRO[0000] error waiting for container.

0 replies

Geremia · 2024-05-28T04:51:38Z

Geremia
May 28, 2024
Author

@wsxiaoys
Disabling userland-proxy (putting "userland-proxy": false in /etc/docker/daemon.json) allows the Tabby container to run, and it says it's listening on port 8080, but still lsof -i :8080 doesn't show anything listening on port 8080.

0 replies

Geremia · 2024-05-28T20:22:08Z

Geremia
May 28, 2024
Author

This was a docker issue. Upgrading docker and docker/cli to from 24.0.2 → 26.1.3 helped

It's listening now:

# lsof -i :8080
COMMAND   PID USER FD   TYPE  DEVICE SIZE/OFF NODE NAME
dockerd 42245 root 26u  IPv4 1047047      0t0  TCP *:8080 (LISTEN)

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GPU Tabby: "Error response from daemon: driver failed programming external connectivity on endpoint" #2265

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 13 comments

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

GPU Tabby: "Error response from daemon: driver failed programming external connectivity on endpoint" #2265

Geremia May 24, 2024

Replies: 13 comments

wsxiaoys May 27, 2024 Maintainer

Geremia May 27, 2024 Author

wsxiaoys May 27, 2024 Maintainer

Geremia May 27, 2024 Author

Geremia May 28, 2024 Author

wsxiaoys May 28, 2024 Maintainer

Geremia May 28, 2024 Author

Geremia May 28, 2024 Author

wsxiaoys May 28, 2024 Maintainer

Geremia May 28, 2024 Author

Geremia May 28, 2024 Author

Geremia May 28, 2024 Author

Geremia May 28, 2024 Author

Geremia
May 24, 2024

wsxiaoys
May 27, 2024
Maintainer

Geremia
May 27, 2024
Author

wsxiaoys
May 27, 2024
Maintainer

Geremia
May 27, 2024
Author

Geremia
May 28, 2024
Author

wsxiaoys
May 28, 2024
Maintainer

Geremia
May 28, 2024
Author

Geremia
May 28, 2024
Author

wsxiaoys
May 28, 2024
Maintainer

Geremia
May 28, 2024
Author

Geremia
May 28, 2024
Author

Geremia
May 28, 2024
Author

Geremia
May 28, 2024
Author