On-call without burnout: escalation policies that respect your team
Good on-call catches incidents fast without wrecking your engineers. Here's how to design rotations, thresholds, and escalation that page the right person at the right time — and no one otherwise.
On-call is where reliability meets human cost. Done well, it catches problems before customers notice. Done badly, it’s a pager that cries wolf at 3 a.m. until your best engineers quietly update their résumés. The difference isn’t how many alerts you send — it’s how carefully you decide what pages whom, and when.
Page on symptoms, not causes
The fastest way to burn out a rotation is to alert on every internal metric. High CPU, a full queue, a slow query — these are diagnostics, not incidents. Page on the handful of signals that mean a user is actually affected: the service is unreachable, error rates are up, checkout is failing. Everything else belongs on a dashboard you look at during business hours, not on the pager.
Confirm before you page
Most 3 a.m. pages that resolve themselves were never real. Two safeguards eliminate the bulk of them:
- Multi-region confirmation — require the failure to be seen from more than one location, so a network blip on one path doesn’t wake anyone.
- Consecutive-failure thresholds — require a check to fail a few times in a row before it counts, so a single dropped connection stays quiet.
Together these turn “something flickered” into “something is genuinely wrong” before a human is involved.
Design the escalation ladder
A good escalation policy assumes the first person might miss it — without blasting everyone at once:
- Alert the primary on-call. Give them a few minutes to acknowledge.
- Escalate to a secondary if there’s no ack. People sleep through phones; the ladder is the backstop.
- Escalate to a lead or wider channel only for the rare unacknowledged, still-firing incident.
The goal is that the right person is reached quickly, and everyone else stays asleep.
Respect the humans in the rotation
- Rotate fairly and keep shifts short enough to recover between them.
- Route by service, so people are only paged for systems they can actually fix.
- Give a quiet default. If a normal night is zero pages, every page carries real signal.
- Review noisy alerts in your retro. An alert that fires often and never matters isn’t monitoring — it’s training your team to ignore the pager.
The bottom line
Sustainable on-call is mostly about restraint: page on user-facing symptoms, confirm before alerting, and escalate through a ladder instead of a firehose. Protect the rotation and you keep both your reliability and the people who deliver it. A pager that only rings when it matters is one your team will actually trust.
Build alerting your team won’t dread. Explore alerting & on-call →
