jupyterhub_traefik_proxy does not upload dynamic toml file to a standalone traefik server

[RusmanCool]

Bug description

I am setting up JupyterHub with a standalone traefik proxy server using both as docker containers. JupyterHub can reach auth-api traefik's port successfully with the given user/pwd. Nevertheless, it does not upload rules.toml file to Traefik proxy when a new user jupyter container is spawned.
If I manually configure rules.toml, traefik picks it up just fine and I can reach Jupyterhub login page and access its dashboard for a provided user. But this is where TraefikTomlProxy fails to upload rules.toml, and the wait time ends and jupyterhub reports an error like: "asyncio.exceptions.TimeoutError: Traefik route for /user/tddd_haha configuration not available"

Some initial observation in the code

1. Take a look at **\jupyterhub_traefik_proxy\toml.py line#190 within add_route() function.
2. Then follow to **\traefik_utils\ persist_routes() function.
   Here there is a single function is called: toml.dump(routes_dict, config_fd) and NO API call to upload rules.toml file to Traefik is made at all. I assume this would work if traefik is managed by Jupyterhub locally/within the same container only.

Expected behaviour

After a new JupyterLab container is spawned, its route is added to Traefik using "file" provider's rules/toml file

Actual behaviour

rules.toml never gets uploaded/updates to/on Traefik

How to reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Your personal set up

• OS:

JupyterHub docker containers based on ubuntu:latest

• Version(s):

versions are whatever latest in pip

• Full environment

running from docker container that installs necessary latest packages in pip

# paste output of `pip freeze` or `conda list` here

• Configuration

# jupyterhub_config.py

# Configuration file for jupyterhub with a full external stack

from jupyterhub.auth import DummyAuthenticator
from dockerspawner import DockerSpawner
from jupyterhub_traefik_proxy import TraefikTomlProxy

c = get_config()

# User containers will access hub by container name on the Docker network
## The ip address for the Hub process to *bind* to.
#  By default, the hub listens on localhost only. This address must be accessible from
#  the proxy and user servers. You may need to set this to a public ip or '' for all
#  interfaces if the proxy or user servers are in containers or on a different host.
#  
#   See `hub_connect_ip` for cases where the bind and connect address should differ,
#   or `hub_bind_url` for setting the full bind URL.
#  Default: '127.0.0.1'
c.JupyterHub.hub_ip = 'jupyterhub'

## The internal port for the Hub process.
#  This is the internal port of the hub itself. It should never be accessed directly.
#  See JupyterHub.port for the public port to use when accessing jupyterhub.
#  It is rare that this port should be set except in cases of port conflict.
#  See also `hub_ip` for the ip and `hub_bind_url` for setting the full bind URL.
##  Default: 8081
c.JupyterHub.hub_port = 8081

## The public facing URL of the whole JupyterHub application.
#  This is the address on which the proxy will bind.
#  Sets protocol, ip, base_url
##  Default: 'http://:8000'
c.JupyterHub.bind_url = 'http://jupyterhub:8000'

#------------------------------------------------------------------------------------
# DockerSpawner settings
#------------------------------------------------------------------------------------
## Use DummyAuthenticator and DockerSpawner
c.JupyterHub.spawner_class = DockerSpawner
## Spawn containers from this image
c.DockerSpawner.image = 'jupyter/base-notebook'
## tell the user containers to connect to our docker network
c.DockerSpawner.network_name = 'jupyterhub'
c.DockerSpawner.use_internal_ip = True
## Pass the network name as argument to spawned containers
c.DockerSpawner.extra_host_config = { 'network_mode': 'jupyterhub' }
## delete containers when the stop
c.DockerSpawner.remove = True
## For debugging arguments passed to spawned containers
c.DockerSpawner.debug = True

#------------------------------------------------------------------------------------
# Routes Proxy settings
#------------------------------------------------------------------------------------
c.JupyterHub.proxy_class = TraefikTomlProxy
# JupyterHub shouldn't start the proxy, it's already running
c.TraefikTomlProxy.should_start = False
c.TraefikTomlProxy.traefik_api_url = "http://traefik:8099"
# traefik api endpoint login username/password
c.TraefikTomlProxy.traefik_api_username = "api_admin"
c.TraefikTomlProxy.traefik_api_password = "api_admin_pwd"
c.TraefikTomlProxy.traefik_log_level = "DEBUG"

#------------------------------------------------------------------------------------
# Authenticator settings
#------------------------------------------------------------------------------------
# dummy for testing. Don't use this in production!
c.JupyterHub.authenticator_class = DummyAuthenticator

#traefik.toml
logLevel = "DEBUG"
debug = true

# the default entrypoint
defaultentrypoints = ["http",]

# the api entrypoint
[api]
	entrypoint = "auth_api"
	dashboard = true

[wss]
	protocol = "http"

[entryPoints]
# the port on localhost where the traefik api and dashboard can be found
	[entryPoints.auth_api]
		address = ":8099"
# authenticate the traefik api entrypoint
	[entryPoints.auth_api.auth]
		[entryPoints.auth_api.auth.basic]
			users = ["api_admin:$apr1$r5/.8rGF$LDX1D/xIuH5YYvWXGiV.z."]
# the port on localhost where traefik accepts http requests
	[entryPoints.http]
		address = ":8000"
# HealthCheck entrypoint
	[entryPoints.hlchk]
		address = ":8089"

# Ping URL: http://hostname:8089/ping to check for Traefik health
[ping]
	entryPoint = "hlchk"

# the dynamic configuration file
[file]
	filename = "/etc/traefik/rules.toml"
	watch = true

#[file]
	#filename = "rules.toml"
	#watch = true

#docker-compose file
version: "3.8" 

services:
    traefik:
        image: traefik:v1.7.34
        container_name: traefik # The service will use this container name.
        restart: unless-stopped
        volumes:
            - ./traefik/traefik.toml:/etc/traefik/traefik.toml
            - ./traefik/rules.toml:/etc/traefik/rules.toml
        ports:
            - "8099:8099"
            - "8000:8000"
            - "8089:8089"
        networks:
            - jupyterhub

    jupyterhub:
        image: jupyterhub-with-dockerspawner-traefik
        container_name: jupyterhub # The service will use this container name.
        restart: "no"
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock:ro # Give access to Docker socket.
            - ./jupyterhub/jupyterhub_config.py:/srv/jupyterhub/jupyterhub_config.py
            #- jupyterhub_data:/srv/jupyterhub
        ports:
            - "8085:8000"
        networks:
            - jupyterhub
        depends_on:
            - "traefik"
            
networks:
    jupyterhub:
        external: true

• Logs

# paste relevant logs here, if any
traefik     | time="2022-02-27T10:42:33Z" level=debug msg="vulcand/oxy/forward/http: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"jupyterhub:8081\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/event-stream\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.9,ru;q=0.8,mt;q=0.7\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Cookie\":[\"jupyterhub-hub-login=\\\"2|1:0|10:1645958552|20:jupyterhub-hub-login|44:ZWRmMTE1NmMwMzk2NDQxMThlNWM0YzkzOWZmYWQ3N2E=|75142655c64bd5bc92832c143e86190e9dca76d0269fb901b5b3000a7474423f\\\"; jupyterhub-session-id=172ed46c313b44498360e1f19d1485f4\"],\"Dnt\":[\"1\"],\"Referer\":[\"http://127.0.0.1:8000/hub/spawn-pending/tddd_haha\"],\"Sec-Ch-Ua\":[\"\\\" Not A;Brand\\\";v=\\\"99\\\", \\\"Chromium\\\";v=\\\"98\\\", \\\"Microsoft Edge\\\";v=\\\"98\\\"\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Windows\\\"\"],\"Sec-Fetch-Dest\":[\"empty\"],\"Sec-Fetch-Mode\":[\"cors\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36 Edg/98.0.1108.56\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"127.0.0.1:8000\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.18.0.1:53256\",\"RequestURI\":\"/hub/api/users/tddd_haha/server/progress\",\"TLS\":null}"
traefik     | time="2022-02-27T10:42:33Z" level=debug msg="Upstream ResponseWriter of type *pipelining.writerWithoutCloseNotify does not implement http.CloseNotifier. Returning dummy channel."
jupyterhub  | [I 2022-02-27 10:42:34.509 JupyterHub dockerspawner:1272] Created container jupyter-tddd-5fhaha (id: fdbb59d) from image jupyter/base-notebook
jupyterhub  | [I 2022-02-27 10:42:34.510 JupyterHub dockerspawner:1296] Starting container jupyter-tddd-5fhaha (id: fdbb59d)
traefik     | time="2022-02-27T10:42:35Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:42:37Z" level=debug msg="Basic auth succeeded"
jupyterhub  | [I 2022-02-27 10:42:38.982 JupyterHub log:189] 200 GET /hub/api (@172.18.0.4) 1.19ms
jupyterhub  | [I 2022-02-27 10:42:39.028 JupyterHub log:189] 200 POST /hub/api/users/tddd_haha/activity ([email protected]) 31.70ms
traefik     | time="2022-02-27T10:42:39Z" level=debug msg="Basic auth succeeded"
jupyterhub  | [W 2022-02-27 10:42:39.625 JupyterHub _version:68] jupyterhub version 2.2.0.dev != jupyterhub-singleuser version 2.1.1. This could cause failure to authenticate and result in redirect loops!
jupyterhub  | [I 2022-02-27 10:42:39.626 JupyterHub base:954] User tddd_haha took 7.397 seconds to start
jupyterhub  | [I 2022-02-27 10:42:39.627 JupyterHub proxy:286] Adding user tddd_haha to proxy /user/tddd_haha/ => http://172.18.0.4:8888
jupyterhub  | [I 2022-02-27 10:42:39.629 JupyterHub proxy:135] Waiting for /user/tddd_haha to register with traefik
traefik     | time="2022-02-27T10:42:39Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:42:39Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:42:40Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:42:40Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:42:41Z" level=debug msg="Basic auth succeeded"
jupyterhub  | [W 2022-02-27 10:42:42.230 JupyterHub base:1054] User tddd_haha is slow to start (timeout=10)
traefik     | time="2022-02-27T10:42:42Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:42:44Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:42:45Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:42:48Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:42:49Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:42:51Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:42:52Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:42:54Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:42:56Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:42:57Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:42:57Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:42:58Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:42:58Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:43:00Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:43:02Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:43:02Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:43:04Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:43:04Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:43:06Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:43:08Z" level=debug msg="Basic auth succeeded"
traefik     | time="2022-02-27T10:43:09Z" level=debug msg="Basic auth succeeded"
jupyterhub  | [E 2022-02-27 10:43:09.871 JupyterHub base:976] Failed to add tddd_haha to proxy!
jupyterhub  |     Traceback (most recent call last):
jupyterhub  |       File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 969, in finish_user_spawn
jupyterhub  |         await self.proxy.add_user(user, server_name)
jupyterhub  |       File "/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py", line 299, in add_user
jupyterhub  |         await self.add_route(
jupyterhub  |       File "/usr/local/lib/python3.8/dist-packages/jupyterhub_traefik_proxy/toml.py", line 204, in add_route
jupyterhub  |         await self._wait_for_route(routespec, provider="file")
jupyterhub  |       File "/usr/local/lib/python3.8/dist-packages/jupyterhub_traefik_proxy/proxy.py", line 150, in _wait_for_route
jupyterhub  |         await exponential_backoff(
jupyterhub  |       File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 189, in exponential_backoff
jupyterhub  |         raise asyncio.TimeoutError(fail_message)
jupyterhub  |     asyncio.exceptions.TimeoutError: Traefik route for /user/tddd_haha configuration not available
jupyterhub  |
jupyterhub  | [E 2022-02-27 10:43:09.882 JupyterHub base:977] Stopping tddd_haha to avoid inconsistent state
jupyterhub  | [I 2022-02-27 10:43:09.908 JupyterHub dockerspawner:1390] Stopping container jupyter-tddd-5fhaha (id: fdbb59d)
traefik     | time="2022-02-27T10:43:10Z" level=debug msg="Basic auth succeeded"

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[alexleach]

I know this is an old question, but I think the issue here is that jupyterhub doesn't have write access to rules.toml. I would recommend mounting the same directory (or volume) in both the traefik and jupyterhub services here.

e.g.

# file: docker-compose.yaml
services:
    traefik:
[...]
        volumes:
            - ./traefik/dynamic-config/:/var/run/traefik/
            - ./traefik/rules.toml:/etc/traefik/rules.toml
[...]
    jupyterhub:
[...]
        volumes:
            - ./traefik/dynamic-config/:/var/run/traefik
            - /var/run/docker.sock:/var/run/docker.sock:ro # Give access to Docker socket.

# file: `traefik.toml`:
[...]
# the dynamic configuration file
[file]
	filename = "/var/run/traefik"
	watch = true

Then, the only other thing you're missing is in your jupyterhub_config.py, which should be told where to write the dynamic configuration. N.B. This API has changed since supporting Traefik v2...

# file: jupyterhub_config.py
# Since supporting Traefik-v2
c.TraefikFileProviderProxy.dynamic_config_file = "/path/to/somefile.toml"

# I'm guessing it would be this configuraiton variable in the version of jupyterhub-traefik-proxy you were using:-
# c.TraefikTomlProxy.dynamic_config_file = "/path/to/somefile.toml"

Hope that helps!