LESSON 22.2
Redundancy, failover and recovery
Remove single points of failure, spread across availability zones, plan recovery with RPO and RTO, and make backups you can actually restore.
5 min read
High availability comes from two habits: don't have a single thing whose failure stops the system, and know exactly how you recover when something fails anyway.
Finding single points of failure
Walk the request path and ask "what if this disappears?" for each component:
- Stateless services: run several instances behind a load balancer, in several zones. Easy.
- Load balancers, DNS, gateways: use managed or redundant pairs.
- Databases: a primary with a standby in another zone and automatic failover.
- Hidden ones: a single cron host, one Redis node, a license server, a config service, the deploy pipeline, one person who knows the system.
Failure domains
Plan capacity for losing one zone: with three zones, each must handle half the load, not a third, when one is gone.
RPO and RTO
Backups only
- Cheap
- Protects against deletes and corruption
- RPO hours
- RTO hours
Pick it when: Internal tools, non-critical data.
Async standby
- RTO minutes
- No write latency penalty
- RPO seconds
Pick it when: Most production databases.
Sync replication
- RPO zero
- Fast failover
- Every write waits for the replica
- Replica outage can stall writes
Pick it when: Payments, ledgers, within one region.
Agree on RPO and RTO per data set with the business; they drive the design and its cost.
Backups that work
Replication copies mistakes instantly: a bad migration or DELETE without WHERE reaches every replica. Backups protect against that.
- Take regular full snapshots plus continuous log archiving, giving point-in-time recovery.
- Store copies in another region and account, with retention locks so an attacker or a mistake can't delete them.
- Test restores on a schedule, and time them. An untested backup has unknown RTO and may not work at all.
Try it
Backup and Restore Platform HLD
Design backup and point-in-time restore for a large database fleet.