two modes:

• compute via traefik API. Only valid if one replica
• compute via prometheus. Required if more than one replica

TODO:

• implement last_activity computation via traefik API (requires implementing simple prometheus collect & sum)
• implement last_activity computation via prometheus query
• decide what to do about websocket activity
• resolve configuration situations for:
  • enable metrics without last_activity
  • enable last_activity with only single endpoint
  • enable last_activity via prometheus (incl auth?)
• tests
• docs

As it currently stands, we have the following configuration:

c.TraefikProxy.enable_last_activity = True

enables single-replica last_activity via the existing traefik API endpoint. _collect_last_activity_api fetches the prometheus metrics endpoint of traefik and parses the relevant metric lines, computing the equivalent of

sum(traefik_router_requests_bytes_total) by (router) + 
sum(traefik_router_responses_bytes_total) by (router)

"activity" is registered as an increase in these values since the last collection (in memory, reset on first collection)

c.TraefikProxy.enable_last_activity = True
c.TraefikProxy.last_activity_prometheus_url = "http://localhost:9090"

requires a prometheus instance configured to collect from traefik, and uses the query:

sum(increase(traefik_router_requests_bytes_total{interval})) by (router) + 
sum(increase(traefik_router_responses_bytes_total{interval})) by (router)

where interval is the number of seconds since the last request. This should produce the same results as above, but handle aggregation across multiple replicas and doesn't require comparison with previous results (increase handles that).

Notably, it seems that websocket messages do not produce measured activity on any traefik metrics: traefik/traefik#10358

Fortunately (or not), JupyterLab still sends tons of API requests even while it is idle, so relying on websocket messages for traffic in 2024 is less critical than it was in the much less resource-intensive days of the classic notebook. But it's still something to be aware of, because active websocket messages are how activity is typically tracked in CHP (e.g. output being produced or execute requests being sent).

Another metric to consider is the open_connections metric, which allows us to consider any open websocket connection to be 'currently active'. This means any open tab (even an idle one) would prevent culling. Sometimes that's good, sometimes it's not, which is why Jupyter Server's internal culler has config to select whether this is considered or not.

closes #151