When caches fail

A cache that absorbs 90% of reads also creates a dependency: the rest of the system is now sized assuming the cache is there and warm. When that assumption breaks, the result is often worse than if there had never been a cache. These failure modes come up constantly in deep-dive questions.

Stampede (thundering herd)

A hot key expires, or is invalidated, and in the next instant thousands of requests miss together. Every one of them queries the database for the same value.

Fixes, from simplest:

Hot keys

Distributed caches shard keys across nodes, so a single key lives on one node. A viral post or celebrity profile can push one node to its limit while the others idle.

• Replicate hot keys across several nodes and read from a random copy.
• Add a tiny local cache in each API server for the hottest keys, with a TTL of a few seconds. Thousands of requests per server become one cache read per few seconds.
• Detect hot keys with sampling, so you can apply these automatically.

Cold starts

A new cache node, a flushed cluster or a failover means an empty cache. The database receives the full read load until the cache warms up. Try it: this design passes at steady load, but the Cold cache scenario empties the cache at 20 seconds. Run it, then click Database and add capacity until the design survives the warm-up.

Adding database capacity works but is expensive to keep idle. Other answers: warm the cache before sending it traffic (replay recent keys), bring nodes back gradually, use coalescing so misses on the same key collapse into one query, and shed load (rate limit or degrade features) rather than letting the database fall over.

The cache is down

What should a read do when the cache times out? Two choices:

Whichever you choose, use short timeouts on cache calls (a few milliseconds), so a sick cache doesn't make every request slow, and a circuit breaker that stops calling it for a while after repeated failures.

Cache penetration

Requests for keys that don't exist always miss, because there's nothing to cache. Bots requesting random IDs can push that load straight to the database. Cache negative results briefly, and for huge key spaces put a bloom filter in front: it answers "definitely not present" for most bogus keys without a lookup.

Quick check

Quick check

Bots request millions of random short codes that were never created. Each one misses the cache and hits the database. What two measures help most?

ALonger TTLs and more cache memoryBBriefly caching 'not found' results and a bloom filter of existing codesCWrite-through cachingDTurning off the cache

Try it

URL Shortener HLD

Your redirect path depends on a cache. Plan what happens when it's cold or gone.