metrics: scope metrics to active config, add optional per-host metrics

[mohammed90]

This PR builds atop @hussam-almarzoq's work in PR #6279, addresses the comments, and scopes the metrics per active config. The admin metrics are retained and not reset per config reload because they're instance/process-oriented. The HTTP metrics are reset at config reload, mostly because it's hard to know the diff to add/remove at reload, especially because radical config change may render previous metrics meaningless.

This also adds 2 more metrics:

• caddy_config_last_reload_success_timestamp_seconds: records the timestamp of the last successful load
• caddy_config_last_reload_successful: records whether the last config load was successful or not

Supersede #6279
Closes #3784

[mholt]

I don't fully understand the metrics aspects, but if this works, that's big if true, yeah?

I will defer to others more proficient with the metrics libs, but if it works and doesn't cause significant regressions in performance, LGTM.

[mohammed90]

I don't fully understand the metrics aspects, but if this works, that's big if true, yeah?

There may be a small uptick in memory utilization, but should be negligible according to the discussion on #4644 (comment). I don't expect the code to be any slower than what's currently experienced in #4644.

[mholt]

Okay. Well I'm approving for the sake of "let's try it" -- not in the sense of "I have read and understand exactly what this is doing" 😅

When someone else with more knowledge approves, then we can merge it in 👍

[hairyhenderson]

I've tested this locally and there are still references to the default registry, especially see https://github.com/caddyserver/caddy/blob/master/modules/metrics/metrics.go#L111 - the effect is that the /metrics endpoint on the admin port exposes the default metrics registered there (Go runtime metrics, for example), but none of the Caddy metrics