Why caches work

Caching is the most common answer to "how do you make this faster and cheaper?", and for good reason. It's also the source of some of the nastiest bugs in production. This module covers both sides. First: why it works at all.

Skew is the reason

If every piece of data were equally popular, a cache holding 1% of it would answer 1% of requests. Real access is skewed: a few links go viral, a few products sell most, a few profiles get most views. Often around 20% of items get 80% of the requests, and the very top items far more. A cache that holds the hot slice answers most requests from memory.

Memory is also simply faster. A read from Redis on the same network takes well under a millisecond. A database query, even a well-indexed one, typically takes a few milliseconds, and much longer when the database is busy.

Hit-ratio arithmetic

Two numbers describe a cache: its hit ratio (the share of reads it answers) and what a miss costs.

The last few percent matter enormously: going from 90% to 99% cuts database load by ten times. That's why caches are sized generously and why techniques that raise hit ratio (longer TTLs, warming popular keys) pay off.

Try it. The cache below starts with a poor hit ratio. Click Redis cache, raise its hit ratio, and run again to watch the database calm down.

Where caches live

A request passes through several places that can cache, and the closer to the user the cache is, the more work it saves.

• Browser and app caches honor HTTP headers like Cache-Control: max-age=3600. Free for you, but you can't invalidate them, so use them for content that rarely changes or is versioned in its URL.
• CDNs cache responses at the network edge, near users. Ideal for images, video, scripts and even public API responses.
• In-process caches in each API server's memory are the fastest, with no network hop, but every server has its own copy, so they're best for small, rarely changing data like configuration.
• Distributed caches (Redis, Memcached) are shared by all servers. This is usually what "add a cache" means in a design.
• The database's own buffer cache keeps hot pages in memory automatically. Size the database's RAM well and many reads never touch disk.

What not to cache

Data that changes on almost every read, data that must always be exact at read time (a bank balance during a transfer), and data that is rarely read twice gain little from a cache and add staleness risk. Cache what is read often, changes rarely, and tolerates being a little stale.