LESSON 28.3
Distributed tracing and sampling
Context propagation across microservices via trace and span IDs, W3C headers, head versus tail sampling, and trace storage.
5 min read
When an e-commerce customer clicks "Buy Now", the request may traverse an edge gateway, authentication service, checkout worker, payment processor, and inventory database. If one of those calls takes three seconds or fails, identifying which service was responsible is nearly impossible without distributed tracing.
Distributed tracing reconstructs the full causal execution path of a transaction across multiple process boundaries.
Context propagation and W3C tracecontext
Distributed tracing works by injecting metadata into outbound requests and extracting it on inbound handlers. The industry standard W3C TraceContext specification uses the traceparent HTTP header:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01- Version (2 hex characters): Current spec is
00. - Trace ID (32 hex characters): A globally unique 16-byte identifier shared across every single microservice participating in this request journey.
- Parent Span ID (16 hex characters): An 8-byte identifier representing the immediate caller's sub-operation.
- Trace Flags (2 hex characters): Controls sampling decisions (e.g.
01means this trace was selected for storage).
The anatomy of a trace
A Trace is a directed acyclic graph (DAG) composed of individual units of work called Spans:
Each span contains:
- Operation name (e.g.
POST /v1/checkout,SELECT FROM inventory). - Start timestamp and duration.
- Key-value attributes (e.g.
http.status_code = 200,db.system = "postgresql"). - Events (such as exceptions, error messages, and log records attached directly to the span).
Head vs tail sampling strategies
At 500,000 requests per second, capturing every span would generate petabytes of telemetry per day. Production systems use sampling to keep costs manageable:
Head-Based Sampling
- Decision is made at the ingress root span before work begins
- Zero collector memory overhead; unsampled requests emit no telemetry downstream
- Blind to outcomes; random 1% sampling will drop 99% of rare errors and slow requests
Pick it when: High-volume uniform traffic where broad statistical trends suffice.
Tail-Based Sampling
- Collector buffers all spans for a request in memory until completion
- Retains 100% of errors (HTTP 5xx) and slow requests (latency > P95) while sampling only 0.1% of successful requests
- Collector nodes require substantial RAM to buffer in-flight spans across long timeouts
Pick it when: Mission-critical user journeys, debugging rare intermittent edge failures.
Trace storage engines
Because spans have unpredictable fields and complex nested query relationships, trace data is typically stored in columnar stores or search engines:
- Jaeger / OpenSearch: Uses inverted indexes on tags and span names for fast visual searching in developer dashboards.
- ClickHouse: Increasingly favored for high-throughput trace storage because columnar compression reduces storage costs by 80% compared to JSON search clusters.
Try it
Distributed Tracing Platform HLD
Design a distributed tracing collector pipeline that ingests, samples, and stores millions of spans per second.