Measuring availability

"Highly available" means nothing until it has a number. Availability is the fraction of time, or of requests, in which the system works well enough for users.

The nines

99%~3.65 days a year

~7.2 hours a month

99.9%~8.8 hours a year

~43 minutes a month

99.95%

~22 minutes a month

99.99%~53 minutes a year

~4.3 minutes a month

99.999%~5 minutes a year

~26 seconds a month

Each extra nine costs far more than the last. At 99.99%, a single human-handled incident uses the whole month's allowance, so recovery has to be automatic.

SLIs, SLOs and SLAs

Measure close to the user (at the load balancer, or with synthetic probes), and measure what users notice: errors and latency on important journeys, not CPU.

Error budgets

An SLO of 99.9% means 0.1% of requests may fail. That 0.1% is the error budget. Spend it on deploys, experiments and planned maintenance. When it runs out, slow down and fix reliability. It turns "is this reliable enough?" from an argument into arithmetic.

Dependencies multiply

So each critical-path dependency must be more available than your target, or redundant, or removed from the critical path (made asynchronous, cached or optional).

Quick check

Quick check

Your checkout page calls a recommendations service that is only 99% available, and checkout's SLO is 99.95%. What's the best design change?

ARaise recommendations to 99.99% firstBMake recommendations optional: call with a short timeout and render checkout without them on failureCLower checkout's SLO to 99%DRetry recommendations until they succeed

Try it

Multi-region Failover HLD

Design a service that survives the loss of a whole region.