Pattern · Scaling
Scaling writes
When writes outgrow one machine. Make writes cheaper, batch and buffer them, partition them, and pick a storage engine built for them.
10 min read
Reads can be copied, cached and served from anywhere. Writes are harder: every write has to land on an authoritative copy, and replicas only multiply them. Scaling writes means doing less work per write, spreading writes across machines, or both.
The problem
A single database primary handles thousands to low tens of thousands of writes a second. Metrics, logs, chat messages, clicks and IoT readings easily exceed that, often in bursts.
Strategies
- Make each write cheaper. Drop unneeded indexes, write narrow rows, avoid read-modify-write round trips, and use upserts.
- Batch. One insert of 500 rows costs far less than 500 inserts. Buffer briefly in the app or a queue and write in bulk.
- Absorb bursts with a queue or log. Accept the write into Kafka quickly, and let consumers write to the database at a steady rate.
- Partition (shard) by key. Spread writes across many primaries so each handles a slice.
- Use a write-optimized engine. LSM-tree stores (Cassandra, ScyllaDB, RocksDB-based systems) turn random writes into sequential appends.
- Aggregate before storing. Store per-minute counts instead of every click when individual events aren't needed.
Choosing a partition key
The key decides whether writes spread evenly:
High-cardinality id (user, device)
- Even spread
- Per-entity locality
- A very active entity becomes a hot shard
Pick it when: Most entity-centric data.
Time
- Easy range scans and expiry
- All current writes hit the newest partition
Pick it when: Only combined with another key, such as (device, day).
Key plus random suffix
- Spreads one hot key
- Reads must gather from every suffix
Pick it when: Known hot keys, such as a celebrity's counters.
Buffering trade-offs
- Buffering in app memory is fastest but loses data on a crash; fine for approximate metrics.
- Buffering in a durable log costs a few milliseconds and survives crashes; right for anything you must keep.
- Asynchronous writes mean readers may not see a write immediately. Tell the client "accepted" (202), not "done", and design the read path for lag.
Common mistakes
- Sharding before trying batching and cheaper writes.
- Partitioning by time alone, creating one hot shard.
- Treating a queue as infinite: if consumers can't keep up on average, the backlog grows forever. Queues absorb bursts, not sustained overload.
- Forgetting that every secondary index multiplies write cost.
Try it
Metrics Ingestion HLD
Ingest millions of metric points a second.