Swarm usb coral device

[hagak]

If this is not appropriate for this group please delete.

So I am using docker swarm and one issue I am running into with swarm is being able to have a container be able to access the usb coral device. I can tag the container to only run on a host with that device however the container gets "operation not permitted" anytime it tries to access the device. Have a workaround by using a docker in a docker which solves this issue. Problem is when you do this I can not figure out how to get traefik to route to this inside container since traefik works on labels and it does not seem to see the labels I put on the internal container. If I put the labels on the outside container traefik routes to the wrong container.
My docker in docker. NOTE enet and mainNetwork are the same.

services:
  iot-ftw:
    image: docker
    command: >-
      docker run --rm --name frigate 
      --network mainNetwork 
      --privileged 
      -v /dev/bus/usb:/dev/bus/usb 
      --mount 'source=frigate_config,target=/config'
      --mount 'source=frigate_clips,target=/media/frigate/clips'
      --mount 'source=frigate_recordings,target=/media/frigate/recordings' 
      --mount type=tmpfs,target=/tmp/cache,tmpfs-size=1000000000
      -v /etc/localtime:/etc/localtime:ro 
      blakeblackshear/frigate:0.8.0-amd64
    networks:
      - enet
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    deploy:
      labels:
         - traefik.http.routers.frigate_main.rule=host(`frigate.home.thetank.org`)
         - traefik.http.routers.frigate_main.entrypoints=https
         - traefik.enable=true
         - traefik.http.services.frigate.loadbalancer.server.port=5000
         - traefik.docker.network=mainNetwork
         - traefik.http.routers.frigate_main.tls.certresolver=myresolver
         - "traefik.http.middlewares.frigate_chain.chain.middlewares=local-network-only,basic-auth"
         - "traefik.http.middlewares.local-network-only.ipwhitelist.sourcerange=127.0.0.1/32, 10.10.0.0/16"
         - "traefik.http.middlewares.basic-auth.basicauth.usersfile=/letsencrypt/users"
         - "traefik.http.routers.frigate_main.middlewares=frigate_chain"
      mode: global

[blakeblackshear]

I'm not very familiar with swarm. I run everything in my house on k3os. It seems like there shouldn't be any reason swarm would need to know anything about the docker inside the container. Did you try running the inside docker in host network mode?

[hagak]

I did try running the inside one as host, however it seems to attach to the HOST network not the network of the parent docker. So this does not provide the isolation I am looking for.

With docker inside docker I can isolate frigate to just my MQTT container and HA container. If i want to see the nice Frigate dashboard I have to expose 5000 on the inner container and that works just has no security on it. IE anyone on the network can see my cameras with no auth. I wanted to put TRAEFIK infront of it to provide TLS and Auth.

[blakeblackshear]

Can you try not mapping the docker.sock into the container? That may prevent the inside container from using the host network for the parent. I don't often run docker inside docker, but I don't think you want to bridge the two together that way.

[hagak]

Tried that, if you do not do that then the container can not access other containers or even DNS of the host. That is pretty much required from what I can see.

[hagak]

Considering switching to K3OS from flatcar only issue is I have a decent number of containers I would have to migrate over and of course have to learn Kubernetes. Probably not a bad thing to learn.