LESSON 28.4
Debugging a slow request
A step-by-step triage playbook for diagnosing tail latency, resource bottlenecks, lock contention, and cascading slow calls.
5 min read
When users complain that "the app is intermittently freezing", finding the root cause can feel like finding a needle in a haystack. P50 median latency might look healthy at 25 ms, while P99 or P99.9 tail latency spikes to 4,000 ms.
Diagnosing tail latency requires a methodical triage playbook that eliminates variables step by step.
The latency triage playbook
- Check the boundaries: Inspect whether latency originated at the client network (slow mobile 3G link), edge CDN, ingress load balancer, or internal application servers.
- Inspect the trace waterfall: Locate the deepest child span that accounts for the majority of the duration. Is time spent executing work, or waiting in a queue?
- Check for missing time (unaccounted gaps): If a parent span took 500 ms but its child spans only took 20 ms, the server spent 480 ms waiting for a worker thread, pausing for garbage collection, or queueing in an internal channel.
- Inspect dependency metrics: Check database connection pool saturation, lock acquisition wait times, and Redis cache hit ratios during the slow window.
- Correlate with resource saturation: Check CPU throttling (common in Kubernetes CFS quotas), memory paging, and network socket packet drops.
Common latency culprits
Database Lock Contention
- Symptoms: Query execution plan is fast in isolation, but takes seconds in production
- Multiple transactions updating the same row or table locks
Pick it when: High-concurrency checkout or inventory reservations.
Garbage Collection Pauses
- Symptoms: All spans across multiple independent threads stall simultaneously on the same host
- Large heaps with millions of ephemeral objects triggering full stop-the-world sweeps
Pick it when: Java / Go services under heavy allocation bursts.
Connection Pool Starvation
- Symptoms: Thread spends hundreds of milliseconds in acquire() before executing a 2ms query
- Pool max size set too low, or slow queries hogging available connections
Pick it when: Relational database connection bottlenecks.
Cascading Fan-out
Tail latency amplification in fan-out calls
In modern distributed architectures, a single web request frequently fans out to hundreds of microservices or storage shards in parallel (for example, fetching recommendations, pricing, inventory, and reviews).
Because the user must wait for the slowest parallel dependency to respond before the page can render, tail latency compounds exponentially:
Probability of tail delay = 1 - (1 - p)^NIf $N = 100$ parallel backend calls and each call has a modest $p = 1%$ chance of experiencing a 500 ms delay, over 63% of user requests will experience the 500 ms delay.
Mitigate fan-out tail latency with:
- Hedged requests: Send an identical secondary request to a different replica if the first request does not respond within the P95 latency threshold.
- Tolerating graceful degradation: If a non-essential service (like personalized badges) exceeds 100 ms, return the page without it.
Numbers to quote
- Human perception of instant
- < 100 ms feels instantaneous to a user
- Acceptable web page P95
- < 500 ms for complete interactive page render
- Database point query budget
- < 5 ms P99 on indexed primary keys
- Internal microservice RPC budget
- < 10 ms P99 for internal service-to-service calls
- Hedged request trigger threshold
- Send a backup call after the P95 latency; cuts the tail sharply for about 5% extra load
Try it
Distributed Tracing Platform HLD
Investigate microservice trace waterfalls and identify the bottleneck slowing down checkout requests.