Live Shard Rebalancing Service · Approach · designlab

Live Shard Rebalancing Service · Approach · designlab

Analysis

Reason about the problem

Approach

Turn it into a plan

Solution

The complete design

01 · Develop the design

Plan the design

Use a durable range-move protocol with one source writer during copying. Version-aware snapshot/log replay prepares the destination, then an enforced source fence and final replay barrier allow a conditional directory ownership switch.

Stage 1

Keep the move state durable

Track preparing, copying, catching-up, fenced, switched and cleaning phases with the range and source/destination epochs.

Stage 2

Budget catch-up capacity

If incoming changes exceed sustained replay capacity, the move cannot catch up without throttling, more capacity or a longer final pause.

Stage 3

Delay source cleanup

Retain forwarding metadata and source data until old requests, retry windows and recovery checks are safely beyond the move boundary.

02 · Work through the method

Step-by-step approach

Reserve the move in an authoritative directory and pin source range epoch, destination placement and a source-supported snapshot/log boundary. Reject concurrent overlapping moves.
Copy snapshot records and tombstones while replaying changes, accepting only newer per-key versions at the destination. Persist progress and keep the destination read/write-inactive for this range.
Fence new source writes and drain already admitted operations to a final log barrier. Replay through the barrier and verify destination contents at that boundary before committing the new directory epoch.
Activate destination requests only under the committed ownership epoch. Old-owner requests receive redirect/retry rather than writes; preserve logical request idempotency across routers. Cleanup follows confirmed cutover and retention horizons, not a timer alone.

03 · Justify the choices

Why this approach works

Can updating router caches alone fence the old owner?

No; servers must enforce the current ownership epoch

A stale router may keep sending writes to the previous owner after directory cutover.

When can destination ownership become active?

After source fencing, final barrier replay and an authoritative directory transition

Copy completion alone does not account for concurrent writes accepted by the source.

How should a delayed source-era update affect a newer copied value?

It must lose under comparable per-key versions and tombstones

Version ordering prevents delayed backfill or replay from undoing a newer value or deletion.

04 · Plan the evidence

Validation strategy

A stale router cannot write through the old owner after cutover.
Deletes racing snapshot copy do not resurrect keys.
Crash in each move phase resumes from durable progress without activating both owners.
Requests retried across the move preserve their logical idempotency identity.

Trade-offs to explain

A final range write pause simplifies single-writer correctness but affects latency.
Versioned copy/replay consumes bandwidth and temporarily duplicates storage.
Directory and server fencing increase coordination while protecting against stale-router split ownership.

Read the detailed solution Build my design