LESSON 28.2
SLIs, SLOs and symptom alerting
Defining actionable service level indicators, budgeting downtime, and alerting on customer symptoms rather than server causes.
5 min read
In distributed systems, servers run hot, disks fill, and individual microservices intermittently throw errors. If an on-call engineer's pager rings every time a single server crosses 90% CPU utilization, the team suffers alert fatigue, leading them to ignore alerts when genuine customer-impacting outages occur.
Modern site reliability engineering focuses on symptom-based alerting built around Service Level Indicators (SLIs) and Service Level Objectives (SLOs).
SLIs, SLOs and SLAs
- Service Level Indicator (SLI): A quantifiable metric that measures the quality of service experienced by users. The two most common indicators are availability (
successful_requests / total_requests) and latency (requests_under_threshold / total_requests). - Service Level Objective (SLO): The target reliability percentage for an SLI measured over a rolling window (typically 30 days). For example, 99.9% of requests must succeed within 300 ms over 30 days.
- Service Level Agreement (SLA): A legal or business contract with customers that stipulates financial refunds or credits if the service breaches a defined threshold. SLAs are always set looser than internal SLOs.
Alert on symptoms, not causes
Cause-Based Alerts (Anti-pattern)
- Easy to configure on infrastructure defaults (CPU > 80%, Memory > 85%)
- Generates false positives when batch jobs run safely
- Fails to alert when a bug returns HTTP 500 while CPU is low
Pick it when: Internal diagnostic dashboards only, never for on-call pagers.
Symptom-Based Alerts (Best Practice)
- Pages only when actual end-user requests are failing or degrading
- Directly aligns engineering alerts with customer pain
- Requires careful metric instrumentation of request endpoints
Pick it when: All production on-call paging policies.
Multi-window burn rate alerts
A naive alert triggers when "error rate exceeds 1% over 5 minutes". This approach has two fatal flaws: a brief 2-minute blip pages an engineer unnecessarily, while a slow 0.5% error rate drains the entire monthly error budget unnoticed.
The standard solution is multi-window burn rate alerting:
- An error budget represents the allowed failure rate (e.g. 0.1% failures for a 99.9% SLO).
- A 1x burn rate consumes exactly 100% of the error budget over 30 days.
- A 14.4x burn rate consumes 2% of the budget in 1 hour (severe emergency; pages on-call immediately).
- A 6x burn rate consumes 5% of the budget in 6 hours (moderate emergency; pages on-call).
- A 1x burn rate alerts a ticketing queue for next-day triage without waking anyone at night.
Numbers to quote
- 99.0% availability
- 7.3 hours downtime per month
- 99.9% availability
- 43.8 minutes downtime per month
- 99.95% availability
- 21.9 minutes downtime per month
- 99.99% availability
- 4.38 minutes downtime per month
- Standard SLO rolling window
- 30 days (720 hours)
Try it
API Gateway HLD
Design an API gateway that monitors service level objectives and protects backends with circuit breaking.