Add metrics endpoint

[lewismiddleton]

I'd like to scrape metrics about the internal runner-manager state. Specifically have it expose how many runners it's tracking in each of its runner groups.

This is useful because we've sometimes observed that AWS EC2 instances may get 'orphaned' meaning the runner-manager service has lost track of it and forgets to delete it. Monitoring these metrics would allow us to detect these 'orphaned' instances by alerting on when the number of EC2 machines != the tracked number of runners.

I propose using Prometheus instrumentation via the prometheus-client library. This will introduce a /metrics endpoint that is used for machine readable metrics.

Are there any objections to this approach?

[tcarmet]

No objections, it would be a very much welcome contribution.

Quick note about the orphan situation, we recently noticed that in our kubernetes manifests we made a mistake on the redis volume path which in result would not persist our data and create orphans every time redis would reboot. (if that's the method of deployment you are using, worth double checking).

[lewismiddleton]

I tried an implementation in #628 but I don't think that's the right approach. To get prometheus metrics, I'm now looking at something like dumping the Redis data as JSON using get_runners() upon request to /groups/{name}/list.

Then writing a custom prometheus exporter to consume that JSON and parse it into prometheus metrics.

[lewismiddleton]

I got something I'm happy with in #630.

#629 is a stand alone change that you may want to keep but I've got what I needed from #630.

[tcarmet]

Looked at both change and I'm happy to get both in. I think they're both useful and look pretty good to me 👌 . Nice work.