Technology · Time-series store
Time-Series Databases
Stores optimized for timestamped metric and event streams. Data models, Gorilla and DoubleDelta compression, retention policies, downsampling rollups, cardinality explosion, and pull vs push collection.
11 min read
A Time-Series Database (TSDB) is a specialized storage engine optimized for sequences of values ordered chronologically: CPU utilization, memory pressure, request latency, stock ticks, and IoT sensor streams. Unlike general-purpose relational databases where timestamps are merely another table column, a TSDB organizes on-disk data structures around time chunks, achieving 90%+ compression and instant range aggregations across billions of samples.
When to reach for one
- Infrastructure and application monitoring (APM): Collecting metrics (latency, error counts, saturation) from thousands of microservices and containers.
- Alerting on operational thresholds: Evaluating rules like "P99 latency exceeding 250 ms for 3 consecutive minutes".
- IoT and hardware sensor monitoring: Ingesting high-frequency telemetry (voltage, temperature, GPS) from millions of distributed devices.
- Capacity planning and historical trends: Graphing traffic growth over months or years.
For full-text log search with string tokens, use Elasticsearch. For high-cardinality multi-dimensional SQL analytics over raw user click events, use ClickHouse.
The time-series data model
Most time-series engines (Prometheus, InfluxDB, TimescaleDB) represent data as discrete series:
series = metric_name + {label_1=val_1, label_2=val_2, ...}Each sample inside a series is an immutable pair: (timestamp, float64_value).
- Labels define identity:
http_requests_total{service="checkout", status="500", env="prod"}is a distinct time series fromhttp_requests_total{service="checkout", status="200", env="prod"}. - Inverted index: The TSDB maintains an inverted index mapping label keys and values to an internal 32-bit Series ID.
- Time chunking: Samples are partitioned into immutable time windows (such as 2-hour blocks in Prometheus or hypertables in TimescaleDB), making bulk deletion by age as simple as dropping an entire disk folder.
Compression: Gorilla and DoubleDelta
Because timestamps arrive at predictable intervals (e.g. every 15 seconds) and metric values change incrementally, TSDBs achieve astonishing compression using techniques pioneered by Facebook's Gorilla paper:
DoubleDelta Timestamp Compression
- Compresses 64-bit timestamps down to an average of 1–2 bits per sample
- Leverages regular collection intervals
- Irregular intervals compress slightly less efficiently
Pick it when: All sequential timestamp streams.
Gorilla XOR Float Compression
- XORs successive 64-bit IEEE floating-point values
- Identical or similar floats compress to 1–4 bits
- Rapidly fluctuating values take more bits
Pick it when: Continuous floating-point metric gauges (CPU, memory, temperature).
Together, these algorithms compress an uncompressed 16-byte sample (int64 timestamp + float64 value) down to 1.37 bytes per sample, allowing a single server to retain billions of data points in memory and local NVMe storage.
The cardinality explosion danger
The single most common operational failure in time-series databases is high cardinality.
Keep metric labels bounded to finite enumerations: service, region, status_code, http_method. If you need to search by individual user ID or order ID, that data belongs in application logs, traces, or ClickHouse.
Retention policies and downsampling rollups
Storing raw 10-second metric samples for two years is wasteful: engineers debugging yesterday's outage need 10-second resolution, but engineers reviewing last year's capacity trend only need 1-hour rollups.
- Raw data tier: Keep raw 10-second or 15-second samples for 7 to 14 days for acute incident triage and high-resolution alerting.
- 5-minute rollup tier: A background rollup job aggregates raw data into 5-minute buckets (storing
avg,min,max,count), retaining this tier for 90 days. - 1-hour rollup tier: Further downsamples into 1-hour aggregates, retained for 1 to 2 years.
- Automated drop: Drop expired raw chunks cleanly without running heavyweight SQL
DELETEqueries.
Pull vs Push metrics collection
Pull Model (Prometheus)
- Central server controls scrape rate and concurrency
- Easy local debugging (curl http://host:9090/metrics)
- Automatic detection of down targets (missing scrape = host dead)
- Requires service discovery to find all targets
- Firewalls and NAT gateways block inbound scrape requests
Pick it when: Kubernetes clusters, container fleets, internal microservices.
Push Model (StatsD, InfluxDB, OpenTelemetry)
- Works seamlessly behind NATs, firewalls, and serverless lambdas
- Short-lived ephemeral jobs push their metrics before exiting
- Fast producers can overwhelm and crash the metrics collector
- Harder to know whether a silent service is healthy or dead
Pick it when: AWS Lambda, mobile clients, edge devices, cross-datacenter telemetry.
Comparing the major players
Prometheus
- Standard for Kubernetes and cloud-native monitoring
- Powerful PromQL language
- Built-in alerting rules
- Single-node design (no built-in clustering; needs Thanos or Cortex for long-term multi-node storage)
Pick it when: Kubernetes cluster observability and alerting.
TimescaleDB
- Built as an extension on top of PostgreSQL
- Full standard SQL, joins with relational business tables
- Automated chunk hypertables
- Higher storage and memory footprint than dedicated Gorilla engines
- Postgres write bottlenecks at very high scale
Pick it when: Teams with Postgres skills who want to join metrics with business data.
InfluxDB
- Purpose-built TSDB with high ingestion performance
- Rich time-series aggregation functions
- TSM engine historically suffered under high cardinality
- Ecosystem churn across versions
Pick it when: IoT fleets, real-time sensor monitoring, standalone dashboards.
Numbers to quote
- Compressed sample size
- 1.3 to 2 bytes per sample (Gorilla + DoubleDelta)
- Ingestion throughput
- 500,000 to 2,000,000 samples/sec per node
- Scrape interval
- 10 to 30 seconds typical for microservices
- Query response time
- < 100 ms for range queries across millions of samples in RAM
- Raw retention window
- 7 to 14 days before downsampling rollups
Where it bites
- Histograms and percentiles: Averaging P99 averages produces mathematically meaningless numbers. Prometheus uses histogram buckets (
le="0.1",le="0.5",le="1.0") to compute percentiles across multiple nodes usinghistogram_quantile(). Sizing bucket boundaries wrong leads to inaccurate tail latency reporting. - Clock drift across monitored hosts: If a host's NTP clock drifts backwards or forwards by several minutes, samples are rejected as out of order or plotted into future time buckets.
- Silent failure on dropped pushes: In push architectures, network partitions drop UDP metric packets silently without producers noticing.
Try it
Metrics Ingestion HLD
Design a real-time metrics ingestion pipeline that handles millions of samples per second without cardinality collapse.