Initializing Systems...

Experiments

We Ran Production on 100% Spot Instances for 30 Days

Everyone said it was reckless. We did it anyway. Here’s the unfiltered data on cost savings, interruptions, and whether we’d do it again.

30 Days
$47K Saved
23 Interruptions
99.91% Uptime
✓ Would Repeat

Spot instances cost 60-90% less than on-demand. The catch? AWS can terminate them with 2 minutes notice whenever they need the capacity back. The conventional wisdom is clear: use spot for batch jobs, never for production. We wanted to see if that wisdom was still correct in 2024.

Spoiler: it’s more nuanced than the hot takes suggest.

The Hypothesis

Our Bet

With proper architecture (stateless services, fast startup, aggressive health checks, and multi-AZ redundancy), spot interruptions become a non-event—just another form of expected failure that the system handles automatically.

If we were right, we’d save roughly $50,000/month. If we were wrong, we’d learn exactly where our architecture was fragile. Either outcome seemed valuable.

Experiment Parameters
Duration 30 days (Nov 1 – Nov 30, 2024)
Scope All stateless services (12 of 17 total)
Excluded Databases, Redis, Kafka, Elasticsearch
Baseline Spend $73,400/month on-demand
Instance Types m5.xlarge, c5.2xlarge, r5.large
Regions us-east-1 (3 AZs)

The Setup

We didn’t just flip a switch. The experiment required three weeks of preparation:

W1

Architecture Audit

Verified all target services were truly stateless. Found 3 services storing session data locally—fixed those first.

W2

Interruption Handling

Built spot interruption handler that catches the 2-minute warning, drains connections gracefully, and triggers immediate replacement.

W3

Capacity Diversification

Configured ASGs to use 8 different instance types as fallbacks. If one type gets reclaimed, others are usually available.

The key architectural change was our spot interruption handler:

spot-handler.py Python
import boto3
import requests
import time

METADATA_URL = "http://169.254.169.254/latest/meta-data"

def check_spot_interruption():
    """
    Poll for spot interruption notice.
    AWS gives us 2 minutes warning—use it wisely.
    """
    try:
        resp = requests.get(
            f"{METADATA_URL}/spot/instance-action",
            timeout=1
        )
        if resp.status_code == 200:
            return resp.json()
    except requests.exceptions.RequestException:
        pass
    return None

def handle_interruption(action_data):
    """
    Graceful shutdown sequence:
    1. Stop accepting new requests
    2. Drain existing connections
    3. Checkpoint any in-flight work
    4. Signal ready for termination
    """
    print(f"⚠️ Spot interruption in {action_data['time']}")
    
    # 1. Deregister from load balancer
    deregister_from_alb()
    
    # 2. Wait for in-flight requests (max 30s)
    drain_connections(timeout=30)
    
    # 3. Flush metrics and logs
    flush_telemetry()
    
    # 4. Notify orchestrator
    notify_replacement_needed()
    
    print("✓ Graceful shutdown complete")

# Main loop - check every 5 seconds
while True:
    interruption = check_spot_interruption()
    if interruption:
        handle_interruption(interruption)
        break
    time.sleep(5)
⚠

The 2-Minute Myth

AWS says you get 2 minutes notice. In practice, we measured 1:47 to 2:03. Don’t cut it close. Our graceful shutdown completes in under 45 seconds, leaving plenty of buffer.

The Data

Here’s what actually happened over 30 days:

💰
$26,100 Actual Spend vs $73,400 on-demand
📉
64% Cost Reduction $47,300 saved
⚡
23 Interruptions ~0.77 per day
⏱️
38 min Total Degradation across all interruptions
Interruption Log (Condensed) 23 total events over 30 days
Date Time Service Instances User Impact Recovery
Nov 3 02:14 api-gateway 2 of 6 None 47s
Nov 5 14:32 search-svc 1 of 4 None 52s
Nov 8 09:47 api-gateway 3 of 6 +120ms latency 1m 34s
Nov 11 16:02 worker-pool 4 of 8 None 2m 12s
Nov 14 11:23 MULTIPLE 7 total 0.3% errors 4m 47s
Nov 17 03:58 image-proc 2 of 4 None 38s
Nov 21 10:15 api-gateway 2 of 6 +85ms latency 1m 02s
… 16 more events (see full log)

The November 14th event was our worst. AWS reclaimed capacity across multiple instance types simultaneously during what appeared to be a regional capacity crunch. For about 5 minutes, we were running at 60% normal capacity.

🔥 Nov 14 Incident Deep-Dive
11:23:14 First interruption notice (api-gateway, 2 instances)
11:23:18 Second notice (search-svc, 2 instances)
11:23:21 Third notice (worker-pool, 3 instances)
11:23:45 Error rate crosses 0.1% threshold
11:24:02 ASG launches replacement instances
11:25:14 All 7 instances terminated
11:26:30 First replacements pass health checks
11:28:01 Full capacity restored, error rate normal
Duration 4m 47s
Requests Affected ~2,400
Failed Requests 847 (0.3%)
Customer Complaints 0

What Surprised Us

Better Than Expected

Interruption Patterns Are Predictable

80% of interruptions happened between 9 AM – 6 PM ET, when on-demand users spin up instances. Nights and weekends were almost entirely interruption-free.

Better Than Expected

Instance Diversity Matters More Than AZs

We thought spreading across 3 AZs would protect us. It helped, but using 8 instance types was more impactful. Reclaims happen by instance type, not location.

Worse Than Expected

Correlated Failures Are Real

The Nov 14 incident taught us that AWS can reclaim across multiple types simultaneously. Our “diverse fleet” wasn’t as diverse as we thought.

Different Than Expected

Savings Vary Wildly by Instance Type

m5.xlarge: 71% discount. c5.2xlarge: 58% discount. r5.large: 82% discount. The “average 60-90%” hides huge variance.

Architecture Changes That Made This Work

01

Capacity-Based Scaling, Not Utilization

We switched from “scale at 70% CPU” to “maintain N+2 capacity always.” This means we’re never running at the edge, so losing 2 instances doesn’t immediately hurt.

desired_capacity = required_capacity + 2
02

Aggressive Prewarming

New instances run a warmup script before joining the load balancer. JIT compilation, connection pool priming, cache population—all before taking real traffic.

Cold start: 4.2s avg response
After warmup: 120ms avg response
03

On-Demand Fallback Pool

When spot capacity is unavailable, ASG automatically falls back to on-demand. We pay full price temporarily but never lose capacity. This kicked in twice during the experiment.

04

Request Hedging

For critical paths, we send the same request to two instances and take the first response. If one instance dies mid-request, the other completes it. Costs 2x compute but eliminates user-visible failures.

The Verdict

✓ We’re Keeping Spot Instances

After 30 days, we’ve made spot instances permanent for all stateless services. The savings are too significant to ignore, and the reliability was better than expected.

But Only Because:
  • All services are truly stateless and can restart cleanly
  • We run N+2 capacity at all times
  • We have automated interruption handling
  • We use 8+ instance types for diversity
  • We have on-demand fallback configured
  • We accept that 1-2 incidents/month will happen
Do This If:
  • Your services are stateless
  • You can tolerate occasional latency spikes
  • You have good health checks and fast startup
  • You’re spending >$20K/month on compute
  • Your team can invest in proper architecture
Skip This If:
  • You have stateful services or long-running jobs
  • You need five-nines reliability guarantees
  • Your startup time is >60 seconds
  • You’re running in a single AZ
  • Your architecture has single points of failure

Key Takeaways

01

Spot Is Production-Ready (With Caveats)

The blanket advice “never use spot for production” is outdated. With proper architecture, spot is a valid choice for stateless workloads. But “proper architecture” is doing a lot of work in that sentence.

02

Instance Diversity > AZ Diversity

Using multiple instance types protected us more than spreading across availability zones. AWS reclaims by instance type, so diversity is your hedge.

03

2 Minutes Is Enough (If You’re Prepared)

The 2-minute warning seemed terrifying before we started. In practice, graceful shutdown takes 30-45 seconds. The real challenge is fast instance replacement, not the shutdown itself.

04

Run the Experiment

We debated this for 6 months before trying it. The actual experiment took 30 days and gave us definitive answers. Stop debating, start testing.

🧪

What Should We Test Next?

We’re always looking for conventional wisdom to challenge. Got an infrastructure assumption everyone accepts but nobody’s tested? Let us know—we might run the experiment.