We Ran Production on 100% Spot Instances for 30 Days
Everyone said it was reckless. We did it anyway. Here’s the unfiltered data on cost savings, interruptions, and whether we’d do it again.
Spot instances cost 60-90% less than on-demand. The catch? AWS can terminate them with 2 minutes notice whenever they need the capacity back. The conventional wisdom is clear: use spot for batch jobs, never for production. We wanted to see if that wisdom was still correct in 2024.
Spoiler: it’s more nuanced than the hot takes suggest.
The Hypothesis
With proper architecture (stateless services, fast startup, aggressive health checks, and multi-AZ redundancy), spot interruptions become a non-event—just another form of expected failure that the system handles automatically.
If we were right, we’d save roughly $50,000/month. If we were wrong, we’d learn exactly where our architecture was fragile. Either outcome seemed valuable.
The Setup
We didn’t just flip a switch. The experiment required three weeks of preparation:
Architecture Audit
Verified all target services were truly stateless. Found 3 services storing session data locally—fixed those first.
Interruption Handling
Built spot interruption handler that catches the 2-minute warning, drains connections gracefully, and triggers immediate replacement.
Capacity Diversification
Configured ASGs to use 8 different instance types as fallbacks. If one type gets reclaimed, others are usually available.
The key architectural change was our spot interruption handler:
import boto3
import requests
import time
METADATA_URL = "http://169.254.169.254/latest/meta-data"
def check_spot_interruption():
"""
Poll for spot interruption notice.
AWS gives us 2 minutes warning—use it wisely.
"""
try:
resp = requests.get(
f"{METADATA_URL}/spot/instance-action",
timeout=1
)
if resp.status_code == 200:
return resp.json()
except requests.exceptions.RequestException:
pass
return None
def handle_interruption(action_data):
"""
Graceful shutdown sequence:
1. Stop accepting new requests
2. Drain existing connections
3. Checkpoint any in-flight work
4. Signal ready for termination
"""
print(f"⚠️ Spot interruption in {action_data['time']}")
# 1. Deregister from load balancer
deregister_from_alb()
# 2. Wait for in-flight requests (max 30s)
drain_connections(timeout=30)
# 3. Flush metrics and logs
flush_telemetry()
# 4. Notify orchestrator
notify_replacement_needed()
print("✓ Graceful shutdown complete")
# Main loop - check every 5 seconds
while True:
interruption = check_spot_interruption()
if interruption:
handle_interruption(interruption)
break
time.sleep(5)
The 2-Minute Myth
AWS says you get 2 minutes notice. In practice, we measured 1:47 to 2:03. Don’t cut it close. Our graceful shutdown completes in under 45 seconds, leaving plenty of buffer.
The Data
Here’s what actually happened over 30 days:
The November 14th event was our worst. AWS reclaimed capacity across multiple instance types simultaneously during what appeared to be a regional capacity crunch. For about 5 minutes, we were running at 60% normal capacity.
What Surprised Us
Interruption Patterns Are Predictable
80% of interruptions happened between 9 AM – 6 PM ET, when on-demand users spin up instances. Nights and weekends were almost entirely interruption-free.
Instance Diversity Matters More Than AZs
We thought spreading across 3 AZs would protect us. It helped, but using 8 instance types was more impactful. Reclaims happen by instance type, not location.
Correlated Failures Are Real
The Nov 14 incident taught us that AWS can reclaim across multiple types simultaneously. Our “diverse fleet” wasn’t as diverse as we thought.
Savings Vary Wildly by Instance Type
m5.xlarge: 71% discount. c5.2xlarge: 58% discount. r5.large: 82% discount. The “average 60-90%” hides huge variance.
Architecture Changes That Made This Work
Capacity-Based Scaling, Not Utilization
We switched from “scale at 70% CPU” to “maintain N+2 capacity always.” This means we’re never running at the edge, so losing 2 instances doesn’t immediately hurt.
desired_capacity = required_capacity + 2
Aggressive Prewarming
New instances run a warmup script before joining the load balancer. JIT compilation, connection pool priming, cache population—all before taking real traffic.
On-Demand Fallback Pool
When spot capacity is unavailable, ASG automatically falls back to on-demand. We pay full price temporarily but never lose capacity. This kicked in twice during the experiment.
Request Hedging
For critical paths, we send the same request to two instances and take the first response. If one instance dies mid-request, the other completes it. Costs 2x compute but eliminates user-visible failures.
The Verdict
After 30 days, we’ve made spot instances permanent for all stateless services. The savings are too significant to ignore, and the reliability was better than expected.
- All services are truly stateless and can restart cleanly
- We run N+2 capacity at all times
- We have automated interruption handling
- We use 8+ instance types for diversity
- We have on-demand fallback configured
- We accept that 1-2 incidents/month will happen
- Your services are stateless
- You can tolerate occasional latency spikes
- You have good health checks and fast startup
- You’re spending >$20K/month on compute
- Your team can invest in proper architecture
- You have stateful services or long-running jobs
- You need five-nines reliability guarantees
- Your startup time is >60 seconds
- You’re running in a single AZ
- Your architecture has single points of failure
Key Takeaways
Spot Is Production-Ready (With Caveats)
The blanket advice “never use spot for production” is outdated. With proper architecture, spot is a valid choice for stateless workloads. But “proper architecture” is doing a lot of work in that sentence.
Instance Diversity > AZ Diversity
Using multiple instance types protected us more than spreading across availability zones. AWS reclaims by instance type, so diversity is your hedge.
2 Minutes Is Enough (If You’re Prepared)
The 2-minute warning seemed terrifying before we started. In practice, graceful shutdown takes 30-45 seconds. The real challenge is fast instance replacement, not the shutdown itself.
Run the Experiment
We debated this for 6 months before trying it. The actual experiment took 30 days and gave us definitive answers. Stop debating, start testing.
Resources
What Should We Test Next?
We’re always looking for conventional wisdom to challenge. Got an infrastructure assumption everyone accepts but nobody’s tested? Let us know—we might run the experiment.