The On-Call Handbook We Wish We Had Three Years Ago
After 1,247 incidents, 3 burnouts, and countless 3 AM pages, we finally built an on-call culture that people don’t hate. Here’s everything we learned.
On-call doesn’t have to be miserable. That’s not wishful thinking—it’s the conclusion we reached after three years of iteration, failure, and finally getting it right. This document is our complete on-call playbook, written for the team we wish we’d been when we started.
Everything here is battle-tested. Some of it is controversial. All of it works for us. Adapt what makes sense, ignore what doesn’t.
1. Our On-Call Philosophy
Before the tactics, the principles. These guide every decision we make about on-call:
On-Call Is Not Punishment
If on-call feels like punishment, something is broken. Either the system is too fragile, the alerts are too noisy, or the compensation is inadequate. Fix the root cause.
Every Page Should Be Actionable
If a human can’t take meaningful action, it shouldn’t page a human. Alerts that require “just check on it” are waste. Delete them or make them actionable.
Sleep Is Sacred
A well-rested engineer at 9 AM is more valuable than a sleep-deprived one at 3 AM. Protect nighttime sleep aggressively. Most things can wait until morning.
You Build It, You Run It
The team that writes the code should be on-call for it. This creates accountability and feedback loops. No “throwing over the wall” to ops.
Incidents Are Learning Opportunities
Every incident reveals something about your system. Blameless post-mortems aren’t just nice—they’re how you prevent the next incident.
Sustainable Beats Heroic
We don’t celebrate firefighting. We celebrate preventing fires. Heroes who work 80-hour weeks during outages are symptoms of systemic failure.
2. The Rotation Structure
We experimented with many rotation structures before landing on this one:
Primary On-Call
- First responder for all pages
- 5-minute response SLA during business hours
- 15-minute response SLA overnight
- Can escalate to secondary after initial triage
Secondary On-Call
- Backup if primary doesn’t respond in 10 minutes
- Available for escalation on complex incidents
- No laptop required, just phone accessibility
- Automatically becomes primary next week
Rotation Rules
- Minimum 4 engineers in rotation (prevents burnout)
- No back-to-back primary weeks
- Skip rotation during PTO (no coverage trading stress)
- New engineers shadow for 2 rotations before going primary
Why Weekly, Not Daily
We tried daily rotations. Context switching every day was exhausting, and nobody built up enough context to handle incidents efficiently. Weekly rotations give you time to learn the current system state.
3. Alert Design Principles
Bad alerts are the #1 cause of on-call misery. Here’s how we design ours:
- API error rate > 50%
- Database primary down
- Payment processing failing
- API latency P99 > 2s
- Error rate 5-50%
- Single replica down
- Background job backlog growing
- Disk usage > 80%
- Cache hit rate dropping
- SSL cert expiring in 30 days
- Dependency update available
- Unusual traffic pattern
The Three Rules of Alert Design
Don’t alert “CPU > 90%”. Alert “API latency > threshold”. Users don’t care about CPU; they care about slow responses.
If you can’t document how to respond, you can’t alert on it. No exceptions.
We review all alerts monthly. Any alert that was acknowledged without action twice gets deleted or fixed.
4. Incident Response Protocol
When the page comes in, this is exactly what happens:
Acknowledge & Assess
- Acknowledge alert in PagerDuty
- Open the linked runbook
- Check the dashboard for scope
- Determine: Is this real? How bad?
Classify & Communicate
- Assign severity (SEV-1 through SEV-4)
- Post in #incidents channel with template
- For SEV-1/2: Start incident Zoom bridge
- Decide: Can I handle alone or need escalation?
Mitigate
- Follow runbook steps
- Focus on mitigation, not root cause
- Update #incidents every 15 min
- Document actions as you go
Verify & Close
- Confirm metrics are back to normal
- Monitor for 15 min to prevent recurrence
- Post resolution summary in #incidents
- Schedule post-mortem if SEV-1 or SEV-2
🚨 **INCIDENT: [Brief Description]** **Severity:** SEV-[X] **Status:** Investigating | Mitigating | Resolved **Impact:** [Who/what is affected] **Started:** [Time] **IC:** @[your name] **Current Understanding:** [1-2 sentences on what we know] **Actions Taken:** - [Action 1] - [Action 2] **Next Update:** [Time, usually 15-30 min]
The #1 Mistake: Hunting Root Cause During Outage
When you’re paged at 3 AM, your job is to stop the bleeding—not solve the mystery. Mitigate first, investigate later. Rolling back a deploy is always faster than debugging it.
5. Runbooks That Actually Get Used
We have 47 runbooks. They actually get used. Here’s why:
Every Alert Links to a Runbook
The alert itself contains the runbook URL. You never have to search for it.
Runbooks Are Tested Quarterly
We run game days where someone follows each critical runbook step-by-step. If it doesn’t work, we fix it.
Runbooks Have Owners
Every runbook has an owner who’s responsible for keeping it current. Ownership rotates with the on-call rotation.
Runbooks Are 3 AM-Friendly
Written for someone who just woke up. No jargon, no assumptions, explicit commands you can copy-paste.
One sentence: what this runbook is for and when to use it.
How to confirm this is the right runbook. Dashboard links, log queries, things to check.
Numbered steps with exact commands. Copy-pasteable. No thinking required at 3 AM.
How to confirm the fix worked. What metrics should look like.
Who to contact if this doesn’t work. Slack channels, phone numbers.
Optional: why this happens, long-term fix ideas. For daytime reading, not 3 AM.
6. Handoff Procedures
A bad handoff sets up the next person for failure. Here’s our handoff ritual:
Outgoing On-Call Prepares:
Incoming On-Call Confirms:
Handoff Timing
7. Compensation & Recovery
On-call is extra work. We compensate for it. Here’s our structure:
Recovery Time
After Overnight Page
If paged between midnight and 6 AM, you can start the next day late. Take the hours you lost to sleep.
After Major Incident
Any SEV-1 lasting >2 hours: take the next day off, or a half day if you prefer. No questions asked.
After Heavy Week
If you had >5 pages in a week, you’re automatically skipped in the next rotation. Someone else takes your slot.
Why We Pay for On-Call
Companies that don’t compensate on-call are implicitly saying the work doesn’t matter. It does. Paying for it also creates organizational pressure to reduce pages—because pages cost money.
8. Continuous Improvement
The system only gets better if you actively improve it. Here’s our process:
Weekly: Alert Review
Every Monday, the outgoing on-call reviews all alerts from the past week. Any alert that fired but required no action gets flagged for deletion or fixing.
After Each Incident: Post-Mortem
SEV-1 and SEV-2 incidents get blameless post-mortems within 48 hours. Action items are tracked in Jira with owners and due dates.
Monthly: Metrics Review
We track pages per week, MTTR, time to acknowledge, and on-call satisfaction. Any negative trends get addressed.
Quarterly: Game Days
We intentionally break things in staging (sometimes production) to test our response. Runbooks get updated based on what we learn.
Key Takeaways
Treat On-Call as a Product
Your on-call experience has users (your engineers). Measure satisfaction. Iterate. Fix pain points. Make it better every quarter.
Fewer, Better Alerts
Delete noisy alerts ruthlessly. Every unnecessary page erodes trust in the system and burns out your team. Quality over quantity.
Compensate the Work
On-call is real work that deserves real compensation. Paying for it also creates healthy pressure to reduce the burden.
Document Everything
Runbooks, handoffs, post-mortems—the teams that document well are the teams that respond well. Make it a habit.
The Goal
On-call should feel like a responsibility you’re prepared for, not a burden you dread. If your engineers are afraid of their on-call week, something is broken. Fix it. They’ll thank you—and they’ll stick around longer.