LESSON 9.2
Failover, durability and backups
Synchronous versus asynchronous replication, what happens when the primary dies, and backups that can actually restore.
6 min read
Machines fail. Disks die, zones lose power, someone deploys a bad migration. A database design isn't finished until you can say what happens to the data, and to users, when that occurs.
Synchronous or asynchronous replication
Asynchronous
- Fast writes: the primary doesn't wait
- Replicas can be far away
- Failover can lose the most recent writes
Pick it when: Most read replicas, cross-region copies.
Synchronous
- A committed write is on at least two machines
- Every write waits for a replica
- A slow replica slows writes
Pick it when: Money, orders and anything where losing an acknowledged write is unacceptable.
Semi-synchronous / quorum
- Wait for one or a majority of replicas, not all
- Still adds latency
Pick it when: A common middle ground in managed databases.
Many managed databases run a synchronous standby in another zone for durability, plus asynchronous read replicas for scale.
Failover
When the primary dies, a replica must take over:
- Detect the failure, typically after several missed heartbeats over 10 to 30 seconds. Too quick and a brief network blip triggers a needless failover.
- Choose the most up-to-date replica.
- Promote it to primary and point clients at it, via DNS, a proxy or a virtual IP.
- Fence the old primary so that if it comes back it can't accept writes. Two primaries accepting writes ("split brain") corrupts data.
Writes fail for the duration, typically tens of seconds to a few minutes. Applications should retry with backoff, and the design should say what users see meanwhile.
RPO and RTO
Two numbers turn "it should be reliable" into a design:
- RPO (data you can lose)
- 0 for payments; minutes for analytics
- RTO (time to recover)
- under a minute for core paths; hours for internal tools
RPO of zero needs synchronous replication. A low RTO needs automated failover with a warm standby. Relaxed targets allow cheaper designs; say which the requirements call for.
Backups
Replication copies mistakes as faithfully as it copies data. A DELETE without a WHERE, or a bug that corrupts rows, reaches every replica in seconds. Protection against that comes from backups:
- Full snapshots taken regularly (daily), stored in another region.
- Continuous archiving of the change log, which enables point-in-time recovery: restore to 10:41:59, one second before the bad migration.
- Restore tests. A backup you have never restored is a hope, not a backup. Measure how long a restore takes, because that's part of your RTO.
Try it
Backup and Restore Platform HLD
Design backups you can actually restore within a target time.