LESSON 28.1
Metrics, logs and traces
The three pillars of observability, how their trade-offs in cardinality, cost and retention balance, and when to reach for each.
5 min read
In a single monolithic server, understanding system behavior is straightforward: you inspect a single log file or attach an interactive debugger. In a distributed architecture with dozens of microservices, asynchronous queues, and database shards, failures are emergent and unpredictable.
Observability is the ability to infer the internal state of a complex system based on its external outputs. Production systems rely on three complementary telemetry data types: metrics, logs, and traces.
The three telemetry signals
Metrics
- Extremely cheap to store (1 to 2 bytes per point)
- Fast real-time aggregation and mathematical alerting
- Retained for years for capacity planning
- Low cardinality; cannot attach high-cardinality IDs like user_id or order_id
Pick it when: Threshold alerting, dashboard graphs, error rate and latency monitoring.
Structured Logs
- Rich, full-context text and attributes for deep forensic debugging
- Contains exact error stack traces and customer identifiers
- Massive disk and network costs; expensive full-text indexing at scale
Pick it when: Post-incident root cause analysis, security audit trails.
Distributed Traces
- Visualizes the exact call graph and latency breakdown of a single request across all microservices
- Requires instrumentation and context propagation; expensive to store 100% of traces
Pick it when: Pinpointing tail latency bottlenecks, diagnosing microservice dependency timeouts.
Triangulating across signals
When an incident strikes, effective engineers correlate all three pillars together:
- Detect with metrics: An alert fires because the
http_request_duration_secondsP99 metric on the checkout service spikes from 50 ms to 3,000 ms. - Isolate with distributed traces: You click through to exemplar traces captured during the spike. The trace waterfall reveals that 2,900 ms of the request was spent inside a downstream call from
CheckoutServicetoInventoryService. - Diagnose with structured logs: You copy the trace ID from the slow span and search your log aggregator (such as Elasticsearch). The logs reveal a database transaction deadlock on the inventory table.
Balancing costs and cardinality
Telemetry pipelines can easily generate more network traffic and data storage than the underlying business application. Keeping telemetry affordable requires clear boundaries:
- Keep metric labels strictly bounded to finite sets (service, region, response code) to avoid cardinality collapse.
- Emit logs in structured JSON format rather than free-form text, enabling fast field filtering without brittle regular expressions.
- Sample distributed traces intelligently so you only retain representative and anomalous requests.
Try it
Metrics Ingestion HLD
Design a real-time metrics ingestion pipeline that handles high-frequency telemetry without cardinality collapse.