Add a way to introspect failing/restarting services

[benhoyt]

Per discussion at #86 (comment), it wasn't obvious when ServiceInfo.Restarts was reset to 0 (if at all), and whether this was the right thing to expose. Knowing whether Pebble has restarted 1000 times vs 1001 isn't particularly useful, for example.

On the other hand, when debugging a f(l)ailing service, we want the user to be able to see how often it's restarting. We kind of want a differential, like "restarts per minute" or "restarts in last 10 minutes". But that might be just as hard to explain, so maybe the absolute number is simplest and the user can see how fast it's going up over by querying over time, or by knowing when Pebble was started.

I lean towards keeping this field -- it seems quite useful to me -- and having it never reset (until Pebble restarts). It would also be exposed in pebble services.

[benhoyt]

During our discussion at the Frankfurt sprint, we discussed a few potential options, leaning towards using the existing "change" system so that things are tracked:

1. Add Restarts and RestartsSpan, giving the number of restarts in the last N minutes (e.g., 6 restarts in the last 1 minute). Implementation: a sliding window of restart times, keeping only those in the last N minutes.
2. Add a LastStart timestamp (or perhaps Uptime) field. Trivial implementation: just store time.Now() whenever a service is started.
3. Add a Pebble "change" object whenever we restart the service (viewable with pebble changes). Restarting a service is a fairly invasive operation, so perhaps this isn't too heavy. Implementation: a new auto-restart change type with a single auto-restart task. In servstate/handlers.go, backoffTimeElapsed would create the auto-restart change and immediately start the task, instead of calling startInternal directly.

Or some combination of the above, for example, I currently lean towards doing both 2 and 3: adding LastStart as well as recording a change.

[benhoyt]

Notes from meeting just now:

• Have a change that's "recovering this service" when it enters the backoff loop, then one task per restart?
• But the last task would stay open (in Doing state), meaning "I'm waiting and monitoring the service". Until the backoff is reset after backoff-limit time and you go Done.
• Alternatively: a single task with a number of restarts and a log? Maybe try them both and see what looks/works better.
• StartTime on the services API would be useful too.

[benhoyt]

Per comment on #117 (comment), we're planning to do this with events (a new concept) instead, and have started spec JU048 to design that.

[benhoyt]

Similar to what we're planning with check failures - comment, where we kind of came full circle back around to Changes and Tasks ... can we figure out a way to do this with Changes and Tasks (but without creating huge numbers of changes in case of a bad failure)?

[benhoyt]

We're going to start with doing this for health checks (spec here). This still seems useful bit is lower-priority.