Rebalancing and query routing

Choosing a partition key is the beginning. Traffic changes, disks fill, and servers need replacing. A useful partitioning design explains both how ownership moves under live writes and how queries find records when the partition key isn't known.

Moving a live partition

Use logical partition ids and a versioned ownership map rather than making callers remember physical addresses forever. A controller plans moves; routers refresh their maps or follow bounded redirects when an owner changes.

A timeout doesn't tell the controller whether cutover committed. Make the move resumable: it checks durable move state and continues rather than starting a conflicting transfer. Don't accept writes independently on both owners unless a specific replication protocol reconciles them.

Rebalancing needs spare capacity

Moving 200 GB at a sustained 100 MB a second takes about 33 minutes before extra work. During that time, both machines still serve users. Throttle copying by disk, network and replication lag, and keep free space for temporary duplicate data.

Balance by bytes and request cost. Two partitions can be equal in size while one serves a hundred times more traffic. Move a few at a time, use hysteresis so partitions don't bounce between machines, and avoid placing all copies in the same failure domain.

Secondary indexes

The primary partition key rarely serves every query. A file store partitioned by owner might also need lookups by checksum or public link.

A global index can map an email to a user id, then route to the user's current partition. Give the index its own partitioning and recovery policy. If email uniqueness is an invariant, an eventually updated index alone can't enforce it; use an authoritative conditional reservation or transaction.

Scatter-gather

A query without a useful routing key sends subqueries to many partitions and merges the results. For top 20 results, each shard can return its local top 20 and the coordinator merges them. Exact aggregate counts still require all relevant shards or a maintained aggregate.

Fan-out increases work and tail latency. At 1,000 user queries a second across 100 shards, the system handles about 100,000 shard requests a second, before retries. Bound concurrency, propagate deadlines and define whether incomplete results are acceptable. Never label a partial count as exact.