The 47-Minute Outage That Started With a Single Typo
How a misplaced comma in a config file cascaded into our longest outage of the year—and the systemic failures that let it happen.
On March 15, 2024, at 2:34 PM EST, our entire platform went down. Every user. Every endpoint. Every service. For 47 minutes, we were a $0/hour company. The trigger? A single comma in the wrong place.
This is our full, unedited post-mortem. We’re publishing it because we believe the industry learns more from honest failure analysis than from success stories. Everything here actually happened.
A developer added a trailing comma to a JSON config file. Our config validation didn’t catch it. The invalid config propagated to all services. Services crashed on startup. Auto-scaling made it worse by spinning up more broken instances. Recovery was delayed because we couldn’t push a fix—our deployment pipeline was also down.
Timeline of Events
Config Change Merged
Developer merges PR #4892 adding a new feature flag. The change includes a trailing comma that makes the JSON invalid. Code review approved without catching it.
CI/CD Pipeline Runs
Pipeline passes. Unit tests pass. Integration tests pass. Config validation step… doesn’t exist.
Deployment Begins
Rolling deployment starts pushing new config to production services.
First Service Crashes
api-gateway fails to parse config on restart. Pod enters CrashLoopBackOff. Kubernetes tries to restart it.
Cascade Begins
More services receive the bad config. auth-service, user-service, and billing-service all crash within 30 seconds.
Full Outage
All user-facing services are down. Monitoring shows 100% error rate. PagerDuty goes nuclear—47 alerts in 2 minutes.
Auto-Scaling Kicks In
Kubernetes sees unhealthy pods and scales up. New pods pull the same bad config and immediately crash. We’re now burning through cloud resources on broken instances.
Incident Commander Joins
On-call engineer escalates to SEV-1. Incident commander begins coordinating response. First hypothesis: “database issue.”
Database Ruled Out
Database is healthy. Team pivots to investigating service logs. Logs show “JSON parse error” but there are thousands of log lines to search through.
Root Cause Identified
Engineer finds the config parse error. Traces it back to PR #4892. The trailing comma is identified.
Can’t Deploy Fix
Team tries to push a fix. Deployment pipeline is also down—it depends on the same config service. We can’t fix the problem using our normal tools.
Manual Intervention Begins
Team decides to manually update config in etcd and force pod restarts. This requires direct cluster access that only 2 people have.
Config Fixed
Valid config pushed directly to etcd. Manual pod deletion begins.
Services Recovering
Services start coming back online. Error rate drops from 100% to 15%.
Full Recovery
All services healthy. Error rate at normal levels. Total downtime: 47 minutes.
The Actual Typo
Here’s the exact change that brought down our entire platform:
See it? Line 5 now has a trailing comma after "ai_recommendations": true, but there’s nothing after it before the closing brace. That’s invalid JSON.
We Don’t Blame the Developer
The developer who wrote this PR did nothing wrong. Humans make typos. That’s expected. The failure is in the system that allowed an invalid config to reach production. Every “human error” is actually a system error waiting to be fixed.
Why It Cascaded
A typo shouldn’t cause a 47-minute outage. Here’s the chain of failures that let it happen:
No Config Validation in CI
Our CI pipeline ran unit tests, integration tests, and security scans. It did not validate that config files were syntactically correct. A 2-second JSON.parse() check would have caught this.
Code Review Didn’t Catch It
Two reviewers approved the PR. Both focused on the logic of the new feature flag, not the syntax of the JSON. Trailing commas are easy to miss in review.
Services Crash Instead of Using Cached Config
When services couldn’t parse the new config, they crashed. They didn’t fall back to the last known good config. They didn’t serve stale data. They just died.
Auto-Scaling Made It Worse
Kubernetes saw unhealthy pods and did exactly what it’s supposed to do: scale up to maintain capacity. But new pods pulled the same bad config and crashed. We were amplifying the failure.
Deployment Pipeline Was Also Down
Our deployment tools depended on the same config service. When the config service crashed, we couldn’t deploy a fix through normal channels. We had to go manual.
Limited Cluster Access
Only 2 engineers had direct etcd access to manually fix the config. One was on vacation. One was in a meeting with phone on silent. 16 minutes were lost waiting for someone with access.
Six failures. Any one of them, if prevented, would have either stopped the outage or dramatically reduced its duration.
Impact Assessment
What We Fixed
In the two weeks following the incident, we implemented the following changes:
Config Validation in CI
Added JSON schema validation to the CI pipeline. Invalid configs now fail the build before they can be merged.
Config Fallback in Services
Services now cache their last known good config. If a new config fails to parse, they log an error and continue running with cached config.
Canary Config Deployment
Config changes now deploy to 5% of pods first. If those pods crash, the rollout stops automatically.
Independent Deployment Pipeline
Deployment tools now have their own config path that’s separate from application config. Broken app config can’t take down deployments.
Break-Glass Access Expansion
Direct cluster access expanded from 2 engineers to 6. Emergency runbook documented for manual config fixes.
Automated Config Rollback
If services start crashing after a config change, automatically revert to the previous config version.
What We Learned
Validate Everything That Can Be Validated
Config files, YAML, JSON, environment variables—if it has a schema, validate it in CI. This is embarrassingly simple to implement and catches entire categories of bugs.
Crash Is Rarely the Right Default
When something unexpected happens, crashing should be the last resort. Log the error. Alert. Serve stale data. Degrade gracefully. Crashing makes small problems into big ones.
Your Recovery Path Can’t Depend on the Thing That’s Broken
If your deployment pipeline depends on the system you’re trying to fix, you don’t have a recovery path. Always maintain an independent way to push critical fixes.
“Human Error” Is a System Error
Blaming the developer who made the typo teaches nothing. The system allowed an invalid config to reach production—that’s where the fix needs to happen.
Failure Modes We Didn’t Know We Had
This incident revealed several failure modes we hadn’t considered:
We assumed config changes were low-risk because they didn’t involve code changes. We were wrong. Config is code. Treat it with the same rigor.
Auto-scaling is supposed to help. But when the problem is “new pods crash immediately,” scaling up just makes things worse faster.
Our deployment pipeline depended on services that could be affected by deployments. Circular dependency = no way out.
Emergency access was too restricted. Security is important, but if only 2 people can fix a production emergency, you’re one vacation away from extended downtime.
You’re Not Alone
Similar incidents have happened at much larger companies. Configuration errors are one of the most common causes of outages:
Config-related outages are so common that Google’s SRE book dedicates an entire chapter to configuration management. We should have read it more carefully.
The Uncomfortable Truth
This outage was entirely preventable. Every fix we implemented afterward could have been implemented before. We knew config validation was a good practice. We just hadn’t prioritized it. Sometimes it takes a $84,000 lesson to move something up the backlog.