Replica placement and failure domains

Consistent hashing decides which node owns a key. Storage systems also need to decide where the copies go, and that decision determines whether a single power outage loses data.

Failure domains

Things fail together. A failure domain is a set of components that a single event can take out:

Replication only helps if replicas don't share a failure domain. Three copies on three hosts in one rack give no protection against that rack's switch failing. The rule is: place each replica in a different domain at the level you want to survive. Usually that means different racks within a zone, or different zones within a region.

Placement rules

Systems encode this as rules. In Ceph, the CRUSH algorithm describes the cluster as a hierarchy (region, zone, rack, host, disk) and a rule such as "pick 3 different racks, then one disk in each". Because CRUSH is a deterministic hash over that map, any client can compute where an object lives without asking a central directory.

Placement groups

Tracking billions of objects individually would be impossible. Instead, objects hash into a fixed number of placement groups (thousands), and groups are placed onto disks. Recovery, balancing and health checks all operate on groups. HDFS blocks, Kafka partitions and DynamoDB partitions play the same role: a unit bigger than one item and smaller than a node.

Recovery after a failure

When a disk dies, every group that had a replica on it is now short one copy. The cluster re-creates those copies elsewhere.

Speed matters because the danger is a second failure while redundancy is reduced. Recovery traffic must be throttled so it doesn't starve user traffic, but kept fast enough that the window stays small.

How likely is data loss?

With random placement over many disks, almost any combination of three simultaneous disk failures will hit some group that had all its replicas on exactly those disks, because there are so many groups. Fewer, larger copysets (limiting which disks can hold replicas together) reduce how often a triple failure loses data, at the cost of losing more when it does. It's a real trade-off that large storage systems tune.