The $50K Bot That Saved Our Team
How we automated 80% of our on-call incidents with a surprisingly simple architecture.
Last year, our on-call rotation was a nightmare. Engineers were getting paged 47 times per week on average. Sleep was a suggestion. Burnout was real. Three people quit citing on-call as a primary factor.
Today, we get about 9 pages per week. Same infrastructure. Same user base. The difference? A bot that cost us roughly $50,000 to build (engineering time + infrastructure) and saves us approximately $340,000 annually in reduced turnover and improved productivity.
Here’s exactly how we built it.
The Problem
We analyzed three months of incident data and found a depressing pattern:
We were waking people up at 3 AM to click buttons that a script could click. That’s not just inefficient—it’s disrespectful to your team.
The Architecture
We called it “OpsBot” (naming things is hard). The architecture is embarrassingly simple:
┌─────────────────────────────────────────────────────────────┐
│ ALERT SOURCES │
│ Datadog │ PagerDuty │ CloudWatch │ Custom Monitors │
└─────────────────────────┬───────────────────────────────────┘
│
▼
┌─────────────────┐
│ Webhook Intake │
│ (Express.js) │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Alert Router │
│ (Rule Engine) │
└────────┬────────┘
│
┌───────────────┼───────────────┐
│ │ │
▼ ▼ ▼
┌────────────┐ ┌────────────┐ ┌────────────┐
│ Auto-Heal │ │ Diagnose │ │ Escalate │
│ Actions │ │ & Enrich │ │ to Human │
└─────┬──────┘ └─────┬──────┘ └─────┬──────┘
│ │ │
▼ ▼ ▼
┌────────────┐ ┌────────────┐ ┌────────────┐
│ Execute │ │ Slack │ │ PagerDuty │
│ Runbook │ │ Summary │ │ Page │
└────────────┘ └────────────┘ └────────────┘
Three paths, three outcomes:
The Rule Engine
The brain of OpsBot is a simple rule engine. Each rule has three parts: a matcher, a confidence score, and an action.
module.exports = {
name: 'high-memory-auto-heal',
// When does this rule apply?
match: (alert) => {
return alert.type === 'HIGH_MEMORY_USAGE'
&& alert.service !== 'database' // Never auto-restart DB
&& alert.value < 95; // Below critical threshold
},
// How confident are we in auto-healing?
confidence: (alert, context) => {
let score = 0.7; // Base confidence
// Boost if we've seen this exact pattern before
if (context.previousOccurrences > 5) score += 0.1;
// Reduce if it's been flapping
if (context.alertsLastHour > 3) score -= 0.2;
// Reduce during business hours (more cautious)
if (isBusinessHours()) score -= 0.1;
return Math.min(Math.max(score, 0), 1);
},
// What do we do if confidence > threshold?
action: async (alert, context) => {
// 1. Take a memory snapshot first
await diagnostics.captureHeapSnapshot(alert.service);
// 2. Gracefully restart the service
await kubernetes.rolloutRestart(alert.service);
// 3. Wait for health check
await health.waitForReady(alert.service, { timeout: 120 });
// 4. Verify memory is actually lower
const newMemory = await metrics.getMemoryUsage(alert.service);
return {
success: newMemory < alert.value,
action: 'rollout-restart',
beforeMemory: alert.value,
afterMemory: newMemory
};
}
};
The confidence threshold is configurable per-team. Our platform team runs at 0.6 (more aggressive automation). Our payments team runs at 0.85 (more cautious). Both are valid.
The "Never Auto-Restart" List
Some things should never be automatically restarted: databases, payment processors, anything with "auth" in the name, and anything currently processing a batch job. We maintain an explicit blocklist that trumps all rules.
When In Doubt: Diagnose
When confidence is below threshold but the alert isn't critical, OpsBot enters diagnosis mode. It gathers everything a human would need to investigate, then posts a summary.
This message gets posted to Slack but doesn't page anyone. The on-call engineer sees it when they wake up and can decide if it needs immediate action.
The key insight: not every alert is urgent. Latency spiked at 3 AM when traffic is 10% of peak? That can wait until morning. The bot helps us separate "needs a human now" from "needs a human eventually."
Runbooks as Code
The real power came when we converted our wiki runbooks into executable code. Here's our disk space runbook:
const runbook = {
name: 'disk-space-cleanup',
description: 'Automated disk space recovery',
steps: [
{
name: 'Identify large files',
action: async (ctx) => {
const largeFiles = await ssh.exec(ctx.host,
'find /var/log -size +100M -type f'
);
ctx.largeFiles = largeFiles;
return { found: largeFiles.length };
}
},
{
name: 'Rotate logs',
action: async (ctx) => {
await ssh.exec(ctx.host, 'logrotate -f /etc/logrotate.conf');
return { rotated: true };
}
},
{
name: 'Clear old temp files',
action: async (ctx) => {
// Only delete files older than 7 days
const deleted = await ssh.exec(ctx.host,
'find /tmp -type f -mtime +7 -delete -print | wc -l'
);
return { deletedCount: parseInt(deleted) };
}
},
{
name: 'Clear Docker artifacts',
condition: (ctx) => ctx.hasDocker,
action: async (ctx) => {
await ssh.exec(ctx.host, 'docker system prune -f');
return { pruned: true };
}
},
{
name: 'Verify space recovered',
action: async (ctx) => {
const newUsage = await metrics.getDiskUsage(ctx.host);
const recovered = ctx.originalUsage - newUsage;
return {
newUsage,
recoveredGB: (recovered / 1024).toFixed(2)
};
}
}
],
rollback: async (ctx, failedStep) => {
// Disk cleanup is generally safe, but log what happened
await slack.post('#ops-alerts',
`Disk cleanup failed at step: ${failedStep.name}`
);
}
};
Every step is logged. Every action is reversible (or at least recoverable). The bot executes the same steps a human would—just faster and at 3 AM without complaining.
The Results
After six months of running OpsBot:
What We Got Wrong
It wasn't all smooth sailing. Here's what we'd do differently:
We launched with auto-healing enabled for everything. Within 48 hours, the bot had restarted our payment service during a legitimate traffic spike. Oops. Start with diagnosis-only mode and graduate to auto-healing gradually.
For the first month, we had no good way to see what the bot had done overnight. Engineers would come in confused about why services had been restarted. Now every action is logged to a dedicated Slack channel and a searchable dashboard.
When a real outage happens, you get 50 alerts in 2 minutes. The bot was trying to auto-heal all of them simultaneously, making things worse. We added rate limiting and "storm mode" that pauses auto-healing when alert volume exceeds a threshold.
Build Your Own
You don't need to build a full OpsBot. Start with these three high-value automations:
When disk usage crosses 80%, automatically run logrotate and clean temp files. Simplest possible automation with immediate payoff.
Don't fix anything—just gather context. When an alert fires, automatically pull recent deployments, related metrics, and similar past incidents. Post to Slack.
When a pod gets OOM-killed, capture a heap dump, restart the pod, and post details. Memory leaks need fixing, but the 3 AM page can wait.
Key Takeaways
Audit Your Alerts First
Before building anything, categorize three months of incidents. You'll likely find that 70%+ don't need a human. That's your automation target.
Confidence Scores > Binary Rules
Don't hard-code "always auto-heal" or "never auto-heal." Use confidence scores that factor in context. The same alert at 3 AM with low traffic is different from noon with high traffic.
Diagnosis Is Underrated
Even if you never auto-heal anything, a bot that gathers context and posts a summary cuts MTTR significantly. Start there.
On-Call Is a Retention Issue
Bad on-call drives away good engineers. The ROI calculation should include reduced turnover, not just time saved. For us, that was the biggest factor.
The Bottom Line
You don't need AI, machine learning, or a dedicated platform team. A webhook, some rules, and the courage to let a script restart your pods at 3 AM. That's it. Your on-call engineers will thank you.