LESSON 15.4
Partitioning by time and tenant
Time-bucketed partitions for events and logs, compound keys that bound partition size, isolating large tenants, and cells that cap the blast radius of any one partition.
6 min read
Hashing a key spreads data evenly, but real data has two shapes that need more care: data that grows forever along time, and tenants whose sizes differ by many orders of magnitude.
Time-bucketed partitions
Events, logs, metrics and messages are written once, read mostly while recent, and deleted after a retention period. Partitioning by time fits all three:
- Writes land in the current bucket (this hour, day or month).
- Queries with a time range touch only matching buckets; the rest are pruned.
- Retention drops whole old buckets, which is instant, instead of running a huge DELETE that bloats the table and the logs.
The catch is that all writes hit the newest bucket. On a single database that's fine. In a distributed store, pure time partitioning sends every write to one node, so time is combined with a hash: partition by (hash(source) mod 32, day), and the day's writes spread across 32 partitions.
Compound keys that bound size
Partitioning by an entity id can still produce a monster partition when one entity is enormous: a group chat with millions of messages, a sensor reporting every second for years, a celebrity's followers. Adding a bucket to the key caps it:
- conversation_id
- simple, but one busy chat grows without limit
- (conversation_id, month)
- bounded per month; recent reads hit one or two partitions
- (conversation_id, message_id)
- tiny partitions, but reading a conversation scatters everywhere
Cassandra and DynamoDB designs use this pattern constantly: keep what's read together in one partition, and bucket it so no partition outgrows a node.
Tenants of every size
SaaS data is usually partitioned by tenant, because almost every query is scoped to one tenant. But tenant sizes follow a power law: thousands of tiny ones and a few huge ones.
Pooled (shared tables, tenant_id column)
- Cheap per tenant
- Easy to add tenants
- Noisy neighbors
- One bug can leak across tenants
Pick it when: Many small tenants.
Dedicated partition or database
- Isolation
- Per-tenant tuning, backups and residency
- Operational cost per tenant
Pick it when: Large or regulated tenants.
Most platforms mix them: pooled partitions for the long tail, with a directory that maps each tenant to its location, so a growing tenant can be moved to dedicated capacity without changing its id.
Cells
Taking this further, a cell is a complete, independent copy of the stack (servers, databases, queues) serving a subset of tenants. A bad deploy, a poison message or an overload affects only one cell's tenants. Cells are partitioning applied to the whole system rather than one database, and they turn "everyone is down" into "5% of customers are down".
Try it
Metrics Ingestion HLD
Choose partition keys for high-rate time-series data.