Design Top-K trending

Advanced55 minBuilds Top-K Trending (Heavy Hitters)Read first: 23.3 Count-min sketch and top-K, 21.3 Stream processing, 30.3 Batch vs streaming, lambda and kappa

STEP 1 OF 13

The brief

You are designing the "Trending" feature of a video site: the top 100 most-viewed videos over the last hour, the last day and the last month, globally and per country. Every home page shows these lists, and they should be at most about a minute behind.

How big it is

Raw view events: about 1 million / second
Batched view requests from players: 2,000 / second
Trending list reads: 20,000 / second
Freshness: about a minute
p99 goal for reads: under 150 ms

Players don't send each view on its own: they batch the views they've seen into one request every few seconds, so 2,000 requests a second carry about a million views.

What makes top K tricky

No single machine can count everything. The event rate and the number of distinct videos are both huge.
Windows slide. "The last hour" changes every minute: new views come in and old ones drop out.
Reads are constant and identical. Everyone in a country sees the same list, thousands of times a second.
Two kinds of correct. The hourly list can be approximate, but monthly counts pay creators and must be exact.

All changes saved

0 components · 0 connections

A blank canvas. A better system.

Add components from the palette, then drag between them to connect the request and event flow.

React Flow

Design Top-K trending

Advanced55 minBuilds Top-K Trending (Heavy Hitters)Read first: 23.3 Count-min sketch and top-K, 21.3 Stream processing, 30.3 Batch vs streaming, lambda and kappa

STEP 1 OF 13

The brief

How big it is

Raw view events: about 1 million / second
Batched view requests from players: 2,000 / second
Trending list reads: 20,000 / second
Freshness: about a minute
p99 goal for reads: under 150 ms

Players don't send each view on its own: they batch the views they've seen into one request every few seconds, so 2,000 requests a second carry about a million views.

What makes top K tricky

No single machine can count everything. The event rate and the number of distinct videos are both huge.
Windows slide. "The last hour" changes every minute: new views come in and old ones drop out.
Reads are constant and identical. Everyone in a country sees the same list, thousands of times a second.
Two kinds of correct. The hourly list can be approximate, but monthly counts pay creators and must be exact.

All changes saved

0 components · 0 connections

A blank canvas. A better system.

Add components from the palette, then drag between them to connect the request and event flow.

React Flow