Build an incident-response runbook your team can follow at 3 a.m.
When a system is down, nobody invents a good process on the spot. A runbook turns panic into steps — here's how to write one people actually use.
The worst time to figure out how you handle incidents is during one. At 3 a.m., half-awake, watching revenue drain, nobody designs a clean process — they thrash. A runbook is the antidote: a short, pre-agreed playbook that tells whoever is on call exactly what to do first, next, and last. Here’s how to write one your team will actually follow.
Start with roles, not steps
The first casualty of an unplanned incident is ownership — everyone assumes someone else is driving. Define three roles up front:
- Incident commander — coordinates, makes calls, keeps the timeline. Not necessarily the person fixing it.
- Responder(s) — the hands on the keyboard investigating and mitigating.
- Communicator — updates the status page and stakeholders so the commander and responders aren’t context-switching.
On a small team one person may wear two hats — but name the hats.
Write the first five minutes in detail
The opening minutes decide how the whole incident goes. Spell them out:
- Acknowledge the alert so escalation stops and everyone knows it’s being handled.
- Declare severity using a simple scale (SEV1 = customer-facing outage, SEV2 = degraded, SEV3 = minor). Severity sets urgency and who to wake.
- Open a channel — a dedicated incident channel or call where all coordination happens.
- Post an initial status update — “we’re investigating,” even before you know the cause.
Everything after this is investigation, which is situational — but the opening is always the same, so script it.
Make mitigation beat diagnosis
The instinct is to find root cause first. Resist it. The runbook should push responders to stop the bleeding — roll back the deploy, fail over, disable the feature flag — before understanding exactly why. Users don’t care why the site is down; they care that it comes back. Root cause is for the postmortem.
Link the tools, don’t describe them
A runbook full of prose nobody reads is useless. Make it a checklist with direct links: the dashboard, the rollback command, the status-page editor, the escalation contacts. The person following it at 3 a.m. should click, not read.
Rehearse it
A runbook that’s never practiced is fiction. Run a lightweight game-day once a quarter — simulate a SEV1 and have someone follow the runbook cold. You’ll find the broken links, the missing access, and the ambiguous steps while it’s safe.
The bottom line
A good runbook assigns roles, scripts the first five minutes, prioritizes mitigation over diagnosis, links the tools directly, and gets rehearsed. It won’t make incidents fun — but it turns a 3 a.m. scramble into a sequence of steps anyone on the team can follow.
Turn alerts into a clean process. Explore incident management →
