Initializing Systems...

Low Risk

The $50K Bot That Saved Our Team

How we automated 80% of our on-call incidents with a surprisingly simple architecture.

Last year, our on-call rotation was a nightmare. Engineers were getting paged 47 times per week on average. Sleep was a suggestion. Burnout was real. Three people quit citing on-call as a primary factor.

Today, we get about 9 pages per week. Same infrastructure. Same user base. The difference? A bot that cost us roughly $50,000 to build (engineering time + infrastructure) and saves us approximately $340,000 annually in reduced turnover and improved productivity.

Here’s exactly how we built it.

The Problem

We analyzed three months of incident data and found a depressing pattern:

73%
of pages were “check and close”
Human looked, saw it was fine, closed the ticket
18%
were resolved with a restart
Service restart, pod bounce, or cache clear
9%
required actual human judgment
Real incidents needing investigation

We were waking people up at 3 AM to click buttons that a script could click. That’s not just inefficient—it’s disrespectful to your team.

Top 10 Alert Types (3-Month Sample)
1 High memory usage 312 pages Automatable
2 Pod restart loop 287 pages Automatable
3 API latency spike 203 pages Partial
4 Database connection pool exhausted 156 pages Automatable
5 SSL certificate expiring 134 pages Automatable
6 Disk space warning 98 pages Automatable
7 Queue backlog 87 pages Partial
8 Failed health check 76 pages Automatable
9 Third-party API timeout 64 pages Partial
10 Anomaly detection trigger 51 pages Manual

The Architecture

We called it “OpsBot” (naming things is hard). The architecture is embarrassingly simple:

OpsBot Architecture
┌─────────────────────────────────────────────────────────────┐
│                      ALERT SOURCES                          │
│  Datadog  │  PagerDuty  │  CloudWatch  │  Custom Monitors   │
└─────────────────────────┬───────────────────────────────────┘
                          │
                          ▼
                 ┌─────────────────┐
                 │  Webhook Intake │
                 │   (Express.js)  │
                 └────────┬────────┘
                          │
                          ▼
                 ┌─────────────────┐
                 │  Alert Router   │
                 │  (Rule Engine)  │
                 └────────┬────────┘
                          │
          ┌───────────────┼───────────────┐
          │               │               │
          ▼               ▼               ▼
   ┌────────────┐  ┌────────────┐  ┌────────────┐
   │ Auto-Heal  │  │  Diagnose  │  │  Escalate  │
   │  Actions   │  │  & Enrich  │  │  to Human  │
   └─────┬──────┘  └─────┬──────┘  └─────┬──────┘
         │               │               │
         ▼               ▼               ▼
   ┌────────────┐  ┌────────────┐  ┌────────────┐
   │  Execute   │  │   Slack    │  │  PagerDuty │
   │  Runbook   │  │  Summary   │  │    Page    │
   └────────────┘  └────────────┘  └────────────┘

Three paths, three outcomes:

🤖
Auto-Heal
Bot executes predefined fix. Human notified after. No page.
🔍
Diagnose
Bot gathers context, posts to Slack. Human reviews when awake.
🚨
Escalate
Bot can’t help. Human gets paged with full context attached.

The Rule Engine

The brain of OpsBot is a simple rule engine. Each rule has three parts: a matcher, a confidence score, and an action.

rules/high-memory.js JavaScript
module.exports = {
  name: 'high-memory-auto-heal',
  
  // When does this rule apply?
  match: (alert) => {
    return alert.type === 'HIGH_MEMORY_USAGE' 
      && alert.service !== 'database'  // Never auto-restart DB
      && alert.value < 95;             // Below critical threshold
  },
  
  // How confident are we in auto-healing?
  confidence: (alert, context) => {
    let score = 0.7;  // Base confidence
    
    // Boost if we've seen this exact pattern before
    if (context.previousOccurrences > 5) score += 0.1;
    
    // Reduce if it's been flapping
    if (context.alertsLastHour > 3) score -= 0.2;
    
    // Reduce during business hours (more cautious)
    if (isBusinessHours()) score -= 0.1;
    
    return Math.min(Math.max(score, 0), 1);
  },
  
  // What do we do if confidence > threshold?
  action: async (alert, context) => {
    // 1. Take a memory snapshot first
    await diagnostics.captureHeapSnapshot(alert.service);
    
    // 2. Gracefully restart the service
    await kubernetes.rolloutRestart(alert.service);
    
    // 3. Wait for health check
    await health.waitForReady(alert.service, { timeout: 120 });
    
    // 4. Verify memory is actually lower
    const newMemory = await metrics.getMemoryUsage(alert.service);
    
    return {
      success: newMemory < alert.value,
      action: 'rollout-restart',
      beforeMemory: alert.value,
      afterMemory: newMemory
    };
  }
};

The confidence threshold is configurable per-team. Our platform team runs at 0.6 (more aggressive automation). Our payments team runs at 0.85 (more cautious). Both are valid.

💡

The "Never Auto-Restart" List

Some things should never be automatically restarted: databases, payment processors, anything with "auth" in the name, and anything currently processing a batch job. We maintain an explicit blocklist that trumps all rules.

When In Doubt: Diagnose

When confidence is below threshold but the alert isn't critical, OpsBot enters diagnosis mode. It gathers everything a human would need to investigate, then posts a summary.

🤖 OpsBot #incidents 3:47 AM
⚠️ API Latency Spike — api-gateway
Severity Medium
P99 Latency 847ms (baseline: 120ms)
Started 12 minutes ago
Affected Endpoints /api/search, /api/products
📊 Auto-Diagnosis
✓ No recent deployments
✓ Database latency normal
✓ No memory pressure
✗ Redis latency elevated (23ms → 156ms)
✗ Elasticsearch cluster yellow
🎯 Likely Cause
Elasticsearch cluster health degraded. Search endpoints depend on ES. Recommend checking ES node status.
View Dashboard ES Cluster Status Page On-Call

This message gets posted to Slack but doesn't page anyone. The on-call engineer sees it when they wake up and can decide if it needs immediate action.

The key insight: not every alert is urgent. Latency spiked at 3 AM when traffic is 10% of peak? That can wait until morning. The bot helps us separate "needs a human now" from "needs a human eventually."

Runbooks as Code

The real power came when we converted our wiki runbooks into executable code. Here's our disk space runbook:

runbooks/disk-space.js JavaScript
const runbook = {
  name: 'disk-space-cleanup',
  description: 'Automated disk space recovery',
  
  steps: [
    {
      name: 'Identify large files',
      action: async (ctx) => {
        const largeFiles = await ssh.exec(ctx.host, 
          'find /var/log -size +100M -type f'
        );
        ctx.largeFiles = largeFiles;
        return { found: largeFiles.length };
      }
    },
    {
      name: 'Rotate logs',
      action: async (ctx) => {
        await ssh.exec(ctx.host, 'logrotate -f /etc/logrotate.conf');
        return { rotated: true };
      }
    },
    {
      name: 'Clear old temp files',
      action: async (ctx) => {
        // Only delete files older than 7 days
        const deleted = await ssh.exec(ctx.host,
          'find /tmp -type f -mtime +7 -delete -print | wc -l'
        );
        return { deletedCount: parseInt(deleted) };
      }
    },
    {
      name: 'Clear Docker artifacts',
      condition: (ctx) => ctx.hasDocker,
      action: async (ctx) => {
        await ssh.exec(ctx.host, 'docker system prune -f');
        return { pruned: true };
      }
    },
    {
      name: 'Verify space recovered',
      action: async (ctx) => {
        const newUsage = await metrics.getDiskUsage(ctx.host);
        const recovered = ctx.originalUsage - newUsage;
        return { 
          newUsage,
          recoveredGB: (recovered / 1024).toFixed(2)
        };
      }
    }
  ],
  
  rollback: async (ctx, failedStep) => {
    // Disk cleanup is generally safe, but log what happened
    await slack.post('#ops-alerts', 
      `Disk cleanup failed at step: ${failedStep.name}`
    );
  }
};

Every step is logged. Every action is reversible (or at least recoverable). The bot executes the same steps a human would—just faster and at 3 AM without complaining.

The Results

After six months of running OpsBot:

Before OpsBot
47 pages per week
23 min avg response time
4.2 hrs weekly on-call burden
3 engineers quit (citing on-call)
→
After OpsBot
9 pages per week
4 min avg response time
0.8 hrs weekly on-call burden
0 engineers quit (citing on-call)
Return on Investment
Development Cost -$50,000 3 engineers × 2 months + infra
Annual Savings: Reduced Turnover +$225,000 ~1.5 fewer departures × $150K replacement cost
Annual Savings: Engineering Time +$115,000 3.4 hrs/week × 50 weeks × $65/hr × 10 engineers
First Year Net +$290,000 Paid for itself in ~2 months

What We Got Wrong

It wasn't all smooth sailing. Here's what we'd do differently:

01 Started Too Aggressive

We launched with auto-healing enabled for everything. Within 48 hours, the bot had restarted our payment service during a legitimate traffic spike. Oops. Start with diagnosis-only mode and graduate to auto-healing gradually.

02 Didn't Track Bot Actions Well

For the first month, we had no good way to see what the bot had done overnight. Engineers would come in confused about why services had been restarted. Now every action is logged to a dedicated Slack channel and a searchable dashboard.

03 Forgot About Alert Storms

When a real outage happens, you get 50 alerts in 2 minutes. The bot was trying to auto-heal all of them simultaneously, making things worse. We added rate limiting and "storm mode" that pauses auto-healing when alert volume exceeds a threshold.

Build Your Own

You don't need to build a full OpsBot. Start with these three high-value automations:

01
Log Rotation on Disk Alerts
Expected Impact -15% pages
Build Time ~4 hours

When disk usage crosses 80%, automatically run logrotate and clean temp files. Simplest possible automation with immediate payoff.

02
Context Enrichment
Expected Impact -30% MTTR
Build Time ~8 hours

Don't fix anything—just gather context. When an alert fires, automatically pull recent deployments, related metrics, and similar past incidents. Post to Slack.

03
Pod Restart on OOM
Expected Impact -20% pages
Build Time ~6 hours

When a pod gets OOM-killed, capture a heap dump, restart the pod, and post details. Memory leaks need fixing, but the 3 AM page can wait.

Key Takeaways

01

Audit Your Alerts First

Before building anything, categorize three months of incidents. You'll likely find that 70%+ don't need a human. That's your automation target.

02

Confidence Scores > Binary Rules

Don't hard-code "always auto-heal" or "never auto-heal." Use confidence scores that factor in context. The same alert at 3 AM with low traffic is different from noon with high traffic.

03

Diagnosis Is Underrated

Even if you never auto-heal anything, a bot that gathers context and posts a summary cuts MTTR significantly. Start there.

04

On-Call Is a Retention Issue

Bad on-call drives away good engineers. The ROI calculation should include reduced turnover, not just time saved. For us, that was the biggest factor.

✓

The Bottom Line

You don't need AI, machine learning, or a dedicated platform team. A webhook, some rules, and the courage to let a script restart your pods at 3 AM. That's it. Your on-call engineers will thank you.