Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Record "recover-service" change when recovering/restarting a service #117

Closed
wants to merge 4 commits into from

Conversation

benhoyt
Copy link
Contributor

@benhoyt benhoyt commented May 2, 2022

Record a "recover" change when recovering a service, and record one task per restart within that.

This will enable debugging/introspection on whether a service is failing and being auto-restarted. You can introspect it with the changes API or the "pebble changes" and "pebble tasks" CLI commands.

The kind (type) of the changes is "recover" and the kind of the tasks is "restart".

Example output while I service is begin recovered (change 3 is not yet ready):

$ pebble changes
ID   Status  Spawn                Ready                Summary
1    Done    today at 16:08 NZST  today at 16:08 NZST  Autostart service "test2"
2    Done    today at 16:08 NZST  today at 16:09 NZST  Recover service "test2"
3    Done    today at 16:09 NZST  -                    Recover service "test2"

After it's recovered the change is marked done/Ready:

$ pebble changes
ID   Status  Spawn                Ready                Summary
1    Done    today at 16:08 NZST  today at 16:08 NZST  Autostart service "test2"
2    Done    today at 16:08 NZST  today at 16:09 NZST  Recover service "test2"
3    Done    today at 16:09 NZST  today at 16:10 NZST  Recover service "test2"

$ pebble tasks 2
Status  Spawn                Ready                Summary
Done    today at 16:08 NZST  today at 16:08 NZST  Restart service "test2"
Done    today at 16:08 NZST  today at 16:08 NZST  Restart service "test2"
Done    today at 16:09 NZST  today at 16:09 NZST  Restart service "test2"

$ pebble tasks 3
Status  Spawn                Ready                Summary
Done    today at 16:09 NZST  -                    Restart service "test2"

Fixes #104

Somehow I'd missed this when I added the state diagram originally.
This will enable debugging/introspection on whether a service is
failing and being auto-restarted. You can introspect it with the
changes API or the "pebble changes" and "pebble tasks" CLI commands.

The kind (type) of the changes is "recover" and the kind of the tasks
is "restart".

Example output while I service is begin recovered (change 3 is not yet
ready):

$ pebble changes
ID   Status  Spawn                Ready                Summary
1    Done    today at 16:08 NZST  today at 16:08 NZST  Autostart service "test2"
2    Done    today at 16:08 NZST  today at 16:09 NZST  Recover service "test2"
3    Done    today at 16:09 NZST  -                    Recover service "test2"

After it's recovered the change is marked done/Ready:

$ pebble changes
ID   Status  Spawn                Ready                Summary
1    Done    today at 16:08 NZST  today at 16:08 NZST  Autostart service "test2"
2    Done    today at 16:08 NZST  today at 16:09 NZST  Recover service "test2"
3    Done    today at 16:09 NZST  today at 16:10 NZST  Recover service "test2"

$ pebble tasks 2
Status  Spawn                Ready                Summary
Done    today at 16:08 NZST  today at 16:08 NZST  Restart service "test2"
Done    today at 16:08 NZST  today at 16:08 NZST  Restart service "test2"
Done    today at 16:09 NZST  today at 16:09 NZST  Restart service "test2"

$ pebble tasks 3
Status  Spawn                Ready                Summary
Done    today at 16:09 NZST  today at 16:09 NZST  Restart service "test2"

Fixes canonical#104
@benhoyt benhoyt changed the title Record "recover" change when recovering/restarting a service Record "recover-service" change when recovering/restarting a service May 9, 2022
@benhoyt
Copy link
Contributor Author

benhoyt commented May 17, 2022

Per discussion with @hpidcock (who had discussed with @niemeyer at the product sprint), we're leaning towards reserving changes and tasks only for user-initiated actions, and we're likely to add an "events" system and use events for this instead. Moving back to draft.

@benhoyt benhoyt marked this pull request as draft May 17, 2022 03:21
@benhoyt
Copy link
Contributor Author

benhoyt commented Jun 20, 2022

Per above comment, we're likely to use events for this instead. Closing for now (can always reopen later if necessary).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a way to introspect failing/restarting services
1 participant