Skip to content

Latest commit

 

History

History
55 lines (34 loc) · 2.78 KB

after_an_incident.md

File metadata and controls

55 lines (34 loc) · 2.78 KB
Raw
cover description
assets/img/covers/resolved.png
Information on what to do after a major incident. Our followup and after action review procedures.

Information on what to do after a major incident. Our followup and after action review procedures.

Followup Actions for Response Roles

In addition to any direct followup items generated from an incident, each of our response roles will have a few standard followup tasks. These are generally lightweight actions that ensure we organize information and followup with customers appropriately.

Steps for Incident Commander

  1. Update the incident in PagerDuty.

    • Group any related incidents under the primary incident.
    • Set the final severity of the incident.
    • Resolve the incident.

  2. Create the post-mortem, and assign an owner to the post-mortem for the incident.

  3. Send out an internal email to the relevant stakeholders explaining that we had an incident, provide a link to the post-mortem.

  4. Occasionally check on the progress of the post-mortem to ensure that it is completed within the desired time frame.

Steps for Deputy

There are no additional steps after an incident is resolved. However the IC may ask for your help with their steps.

Steps for Scribe

  1. Review the chat communications and extract any relevant items from key events.

  2. Collect all TODO items and add them to the post-mortem.

Steps for Subject Matter Experts

  1. Add any notes you think are relevant to the post-mortem.

Steps for Customer Liaison

  1. Reply to any customer enquiries we received about the incident.

  2. Follow the post-mortem progress, and update our status page with the external message once it is available.

Steps for Internal Liaison

There are no additional steps after an incident is resolved. However the IC may ask for your help with answering questions from internal stakeholders.

Reviewing the Incident

It's important that we review the incident in detail to see exactly what went wrong, why it went wrong, and what we can do to make sure it doesn't happen again. These take many names; after-action reviews, incident review, followup review, etc. We use the term post-mortem.

You can read all about our post-mortem process, which goes over this in more detail.

Reviewing the Process

As well as reviewing the incident, it's important to review our process. Did we handle the incident well, or are there things we could have done better?

This review isn't very formal yet, and typically involves a few of the incident commanders getting together to discuss how we might have done things differently, or if there are any tweaks we can make to our incident response process.

If you're interested in joining these meetings, just let one of the incident commanders know and we'll be sure to invite you.