Testing for failure

Redundancy, replicas and failover plans are hypotheses until they've been exercised. The previous lessons designed for failure; this one is about proving the design works before a real outage does it for you.

Drills

The most valuable tests are simple and recurring:

Each drill tends to find something: a TTL longer than expected, a replica too small to take the load, a credential that only exists in one region, a runbook step that no longer works.

Chaos engineering

Chaos engineering injects faults deliberately to test assumptions continuously, not just during drills. Netflix's Chaos Monkey randomly terminated production instances, which forced every service to tolerate losing one.

A sound experiment has:

• A steady-state metric that defines normal: orders per minute, p99 latency, error rate.
• A hypothesis: "if one cache node dies, error rate stays under 0.1%".
• A small blast radius: one instance, one zone, a small share of traffic.
• An abort condition: stop automatically when the metric moves too far.

Typical faults to inject: killed instances, added latency, dropped network packets, full disks, clock skew and failing DNS.

Game days

A game day is a scheduled exercise where a team runs a realistic scenario ("the primary region is unreachable"), follows its runbooks and measures detection time, decision time and recovery time. It tests people and processes as much as software: who gets paged, who decides to fail over, how customers are told.

Runbooks and on-call

When something breaks at 3 a.m., the on-call engineer needs a runbook: symptoms, dashboards to check, safe actions and when to escalate. The best runbook steps become automation over time. Alerts should be few and actionable, tied to user-visible symptoms (SLO burn) rather than every CPU spike.

Learning from incidents

After an incident, a blameless review asks what happened, why the system allowed it, and what would stop it happening again. The output is concrete follow-ups: a guardrail on a dangerous command, an alert that would have caught it sooner, a load test for the path that failed. Tracking those to completion matters more than the document.