LESSON 17.4
Overload, cascades and graceful degradation
How one slow dependency takes down a whole system, metastable failures that persist after the trigger is gone, thundering herds and cold caches, and degrading features instead of failing everything.
6 min read
Most large outages aren't a single component failing. They're a small problem that spreads: one slow dependency, one expired cache key, one retry loop, until the whole system is overloaded by work it created itself.
How cascades spread
A typical chain:
- A database gets slower, perhaps from a bad query or a failover.
- API servers wait longer for it, so more requests are in flight at once and thread pools and connection pools fill up.
- Upstream callers time out and retry, multiplying the load.
- Health checks start failing on overloaded servers; the load balancer removes them, and the rest get even more traffic.
- Services that call the API, but don't need the database, fail too because the API has no free threads.
The tools from earlier in this module (timeouts, retry budgets, circuit breakers, bulkheads) each break one link in that chain. Use them together.
Metastable failures
Sometimes the system doesn't recover when the trigger goes away. It has a healthy state and an overloaded state, and both are stable:
- Retries
- each failure adds more attempts, so load rises as errors rise
- Work on dead requests
- servers finish requests whose clients already gave up
- Cold caches
- a restart empties caches, so the database gets more load than before
- Expensive error paths
- logging, stack traces and reconnects cost more than successes
The fix is to make the healthy state the only stable one: cap retries, drop requests whose deadline has passed before doing the work, shed load early, and bring capacity back gradually rather than all at once. Often the only immediate remedy is cutting incoming load, which is why systems need a switch to do it.
Thundering herds
Many clients doing the same thing at the same moment:
- Cache stampede: a hot key expires and every request recomputes it. Use request coalescing (single flight), refresh hot keys early in the background, and add jitter to TTLs.
- Synchronized retries or reconnects: add jitter to backoff.
- Cron at midnight: everyone's scheduled job runs at 00:00. Spread start times.
Degrade gracefully
Not every feature matters equally. Decide in advance what the system does when each dependency fails:
Serve stale
- Users see something reasonable
- Data may be old
Pick it when: Feeds, catalogs, prices that can be a minute old.
Drop the feature
- Core page still works
- Less functionality
Pick it when: Recommendations, reviews, badges, personalization.
Queue for later
- No work is lost
- Delayed effect
Pick it when: Emails, analytics, non-urgent writes.
Fail the request
- Honest
- User sees an error
Pick it when: Payments and other operations that must not guess.
Some teams go further with load shedding by priority: under overload, drop background and low-priority traffic first (prefetches, analytics, batch jobs), so checkout and login keep working.
Try it
Flash Sale Admission HLD
Plan how the system degrades when a sale brings ten times the expected traffic.