LESSON 8.3
Capacity, headroom and autoscaling
How many servers you need, why you never run them near full, and why autoscaling alone doesn't save you from a sudden spike.
7 min read
"How many servers do we need?" sounds like an estimation question, and it is, but the interesting part is everything around the arithmetic: headroom, failures, and spikes that arrive faster than any autoscaler.
The arithmetic
- Peak load. From the requirements: say 12,000 requests a second at peak, not the daily average.
- Per-server capacity. Measured by load testing, or estimated: a simple API server might handle 1,000 to 5,000 requests a second.
- Target utilization. Usually 50–70% at peak, not 100%.
- Divide, then add for failures. 12,000 ÷ (1,000 × 0.6) = 20 servers, plus enough to lose a zone.
Why not run at 100%?
A server's response time is its work plus the time a request waits in line. As utilization rises, the line grows, and it grows non-linearly:
- 50% busy
- wait ≈ 1× service time
- 70% busy
- wait ≈ 2.3×
- 90% busy
- wait ≈ 9×
- 95% busy
- wait ≈ 19×
That's why p99 latency falls apart long before a server is "full", and why capacity plans target 50–70%.
The simulator shows it. This render service starts with too few instances. Click it, raise the instances until steady peak passes, then switch the scenario to Spike 3× and see how many more you need.
Try it under load
Plan for failure, not just load
If you run in three availability zones and one goes down, the other two must carry everything. Twenty servers spread over three zones lose about seven at once. Either run enough that two zones can absorb the peak, or accept degraded service during a zone outage, and say which.
Autoscaling
Autoscalers add instances when a metric (CPU, requests per server, queue depth) crosses a threshold and remove them when it falls. They save a lot of money on daily traffic curves. But they react after the load arrives, and new instances take time: minutes to boot, start and warm caches.
Reactive autoscaling
- Cheap
- Follows gradual daily curves well
- Minutes behind sudden spikes
Pick it when: Normal, gradual traffic changes.
Scheduled or predictive scaling
- Capacity is ready before the spike
- Needs a known schedule or a good forecast
Pick it when: Launches, sales, sporting events, the morning peak.
Load shedding and queueing
- Protects the system during the gap
- Some users wait or get errors
Pick it when: Unexpected spikes; a backstop for everything else.
Try it
Flash Sale Admission HLD
Design for a spike that arrives faster than autoscaling can react.