Choosing a partition key

One database eventually runs out of disk space, write throughput or room for useful indexes. A bigger machine buys time, and read replicas help reads, but neither divides the primary write workload. Partitioning, often called sharding across machines, gives different servers responsibility for different subsets of the data.

Split for a reason

State the constraint first. Is the data too large, are writes saturating one server, or does one tenant consume everyone else's capacity? Partitioning adds routing, rebalancing and cross-partition operations. Don't pay those costs before a simpler database can meet the requirement.

Partitioning and replication solve different problems. Four partitions with three copies each means twelve partition copies, not twelve disjoint pieces. Each partition still needs recovery, backups and enough capacity to survive a replica failure.

Range or hash?

Range boundaries might assign customer ids 1–100,000 to one partition and 100,001–200,000 to another. Split growing ranges as needed. With hashing, compute a stable hash and map it to a logical bucket whose owner comes from a routing table. Keeping buckets separate from physical machines makes later moves easier.

An increasing timestamp alone puts every new write in the newest range. A composite key such as (tenant_id, day, bucket) can spread writes while keeping some locality, but queries now need the relevant buckets. The key is a query design decision as much as a storage decision.

Choose from the access pattern

For a file service, partitioning by owner can keep a user's directory listing local. Partitioning by file id distributes files well but makes "list this user's files" depend on another index. Neither is universally better.

Hot keys and hot partitions

Imagine 100,000 writes a second spread across ten partitions. The average is 10,000, but one tenant producing 40,000 writes a second already exceeds that average on its own. Adding partitions won't divide that tenant if the key remains tenant id.

For divisible workloads, add a bucket or salt. Split a tenant's independent events across eight buckets, perhaps using hash(event_id) % 8. Its 40,000 writes become roughly 5,000 per bucket under an even distribution. Reads covering the tenant must now merge eight streams.

Read-heavy hot objects can use cache copies or read replicas. A single mutable balance requiring serial updates can't simply be salted without changing the consistency model; use atomic operations, explicit ownership or a redesigned invariant instead.

Quick check

Quick check

A system hashes file ids evenly, but most requests list all files belonging to one owner. What is missing from the design?

AA longer file idBAn owner-based access path, such as an owner-partitioned index, with a stated update policyCA random timeoutDMore copies of the routing table

Try it

File Storage HLD

Choose a partition key for file metadata and explain how a very large owner is handled.