Metrics Ingestion · Analysis

01 · Frame the problem

Understand the central challenge

Design a multi-tenant metrics service. Producers send batches, clocks may be skewed and a single label can create millions of series. Queries need recent aggregates and explicit retention.

Throughput is constrained by bytes, series cardinality and indexing work, not only samples per second. Define durable acceptance, duplicate sample semantics and how far query visibility may lag ingestion.

02 · Find the design pressure

Start simple, then identify the gap

Begin with validated tenant batches appended to durable storage. Synchronous indexing couples acceptance to query-engine health. A durable log separates those stages, while cardinality quotas prevent small requests from creating unbounded series metadata.

Validate tenant-scoped batches, append them to a durable partitioned log and asynchronously index time series. Quotas, deterministic sample identities and retention control the service’s operating envelope. Authenticate the tenant and bound request bytes, sample counts, label counts and timestamp skew. Canonicalize labels into a tenant-scoped series identity and enforce active-series admission quotas. Append accepted batches durably, partitioned by series identity for useful local ordering. Acknowledge append success and return retryable failure when admission or storage is unavailable. Workers upsert samples using (tenant, series, timestamp). For this exercise, accept an identical duplicate and reject conflicting values for the same identity. Commit indexing progress only after durable writes; replay is then safe. Query by tenant, bounded time range and selected series. Build rollups with idempotent bucket replacement and a defined lateness window. Retain raw and rolled-up data under separate policies and expose indexing lag.

03 · Define success

Requirements to preserve

Authenticate tenants and validate bounded metric batches.
Acknowledge only after durable append.
Handle retries, late samples and replay under a stated duplicate policy.
Enforce cardinality quotas and retention before resource exhaustion.

Workload: 5,000 reads/s, 50,000 writes/s, 100 B records, 300 ms read p99 target. These are exercise assumptions; the Solution includes capacity formulas.

04 · Identify the correctness boundary

Consistency, ownership and failure boundaries

Canonical label ordering defines stable series identity. Limit both request volume and active-series cardinality: a tenant can exhaust memory with many distinct label combinations even at a moderate sample rate. Acknowledge durable log acceptance separately from asynchronous query visibility.

05 · Make it concrete

Reason through a worked example

Two batches contain the same tenant, series and timestamp. Apply the documented duplicate/conflicting-value policy consistently during replay. A batch introducing millions of unique user-ID labels is rejected or quota-limited before unbounded index allocation, while other tenants retain their ingestion budget.