Hot Keys and Partition Skew

Sharding, partitioning, and consistent hashing are designed to distribute keys uniformly across a cluster. However, real-world user traffic follows a power-law (Pareto) distribution: a single key—such as a viral tweet, breaking news alert, high-profile flash sale item, or celebrity live stream—can suddenly receive 100,000+ requests a second.

Even in a 1,000-node cluster, all requests for that key land on the single node owning that key's partition, exhausting its CPU and network NIC while the other 999 nodes sit idle.

The problem: hot read keys vs hot write keys

Detecting hot keys

You cannot mitigate hot keys unless you can identify them before they crash nodes:

• Broker / Store-side LFU tracking: Redis supports redis-cli --hotkeys using Least Frequently Used (LFU) logarithmic counters.
• Client-side sampling: API gateways track request frequencies using probabilistic data structures like a Count-Min Sketch or Top-K algorithm, identifying hot keys in real time without storing every key in memory.
• Cluster skew metrics: Alert when the standard deviation of CPU utilization or network bandwidth across cluster nodes exceeds 30%, signaling partition skew.

Defending hot read keys

1. In-process L1 caching

The most effective weapon against a hot read key is absorbing requests before they ever reach the network:

Laying out the diagram…

• Each API instance maintains a tiny in-process LRU cache (e.g. Caffeine in Java, memory cache in Go/Node.js) with a short TTL (such as 2–5 seconds).
• If 100 API servers receive 100,000 requests/second for a viral post, each API server fetches the post once every 3 seconds. The distributed cache cluster sees only ~33 queries/second instead of 100,000.

2. Request coalescing (Single-flight)

When an extremely hot key expires from the cache, thousands of concurrent requests miss at the exact same millisecond:

• Use a single-flight mutex on the API server.
• The first thread to miss acquires the lock and queries storage.
• All subsequent concurrent threads wait on the same promise/channel and receive the identical result when the first thread returns, turning 5,000 misses into a single database read.

3. Key replication with random read routing

If an asset must be read from the shared distributed cache rather than local memory:

• Replicate the hot key across $N$ distinct keys: item:42#1, item:42#2 ... item:42#16.
• Because consistent hashing hashes the full key string, these 16 keys hash to 16 completely different physical cache servers.
• When reading, the client randomly picks an integer $i \in [1, 16]$ and requests item:42#i. Traffic spreads evenly across 16 machines.

Defending hot write keys

When thousands of requests want to update the same key per second (such as incrementing a view counter or decrementing flash sale inventory), single-node row locks will stall:

1. Key splitting / Sharded counters

A single Redis INCR or SQL row update tops out at a few tens of thousands of writes per second before saturating a single CPU core:

2. Flash sale inventory: admission control and pre-allocation

In a flash sale where 100,000 users attempt to purchase 1,000 phones in the first second:

• Do not let 100,000 requests contend on the same database row.
• Put a virtual waiting room (rate limiter / token bucket) in front of the checkout API, admitting only as many users per second as the inventory service can process.
• Store inventory stock in Redis and decrement with an atomic Lua script (if stock > 0 then stock = stock - 1 return true).

Real-world scenarios

Numbers to quote

Single Redis node throughput

~50,000 to 100,000 ops/sec per core

Single SQL row update limit

~1,000 to 3,000 writes/sec before row lock queue builds

L1 cache TTL

1 to 5 seconds (eliminates 99% of hot key traffic)

Sharded counter shards

10 to 50 sub-keys sufficient for millions of writes/min

Count-Min Sketch memory

a few megabytes to track top-K keys across millions of requests

Where it bites

• L1 cache staleness bugs: If an admin bans a user or changes a product price, and API servers cache the old record in process memory for 60 seconds, users will see the old price. Keep L1 TTLs very short (1–3 seconds) or build a pub/sub invalidation bus for instant local evictions.
• Write amplification of replicated keys: Replicating a hot read key across 16 nodes spreads reads, but every mutation must now write to all 16 keys. Only replicate keys whose read-to-write ratio exceeds 100:1.
• Hot shard migration storms: When an automated cluster manager detects a hot shard and attempts to rebalance/migrate it live during peak load, the migration process itself consumes I/O and network, turning a slow node into an unresponsive node.

Try it

Distributed Cache HLD

Design a distributed cache that handles extreme hot keys and partition skew without node failure.