Design a web crawler

Advanced60 minBuilds Web CrawlerRead first: 10.1 Why queues, 7.1 Why caches work, 17.1 Timeouts and retries

STEP 1 OF 13

The brief

You are designing a distributed web crawler for an approved set of target domains. The crawler consumes seed URLs, crawls discovered pages up to configured depth limits, and persists parsed text and metadata for indexing.

Web crawling looks simple until you meet the messy realities of the public internet:

Cycles and spider traps: Dynamic links and session identifiers create infinite graphs. A crawler that does not deduplicate URLs will crawl forever.
Politeness: If 50 distributed workers hammer one small website simultaneously, they will take it down. The crawler must strictly limit requests per host.
External latency: Fetching remote web pages over DNS and TLS takes hundreds of milliseconds, dwarfing internal network latency.
Worker crashes: Fetchers crash or hang on malicious pages. Leased URLs must be safely timed out and recovered.

All changes saved

0 components · 0 connections

A blank canvas. A better system.

Add components from the palette, then drag between them to connect the request and event flow.

React Flow

Design a web crawler

Advanced60 minBuilds Web CrawlerRead first: 10.1 Why queues, 7.1 Why caches work, 17.1 Timeouts and retries

STEP 1 OF 13

The brief

Web crawling looks simple until you meet the messy realities of the public internet:

Cycles and spider traps: Dynamic links and session identifiers create infinite graphs. A crawler that does not deduplicate URLs will crawl forever.
Politeness: If 50 distributed workers hammer one small website simultaneously, they will take it down. The crawler must strictly limit requests per host.
External latency: Fetching remote web pages over DNS and TLS takes hundreds of milliseconds, dwarfing internal network latency.
Worker crashes: Fetchers crash or hang on malicious pages. Leased URLs must be safely timed out and recovered.

All changes saved

0 components · 0 connections

A blank canvas. A better system.

Add components from the palette, then drag between them to connect the request and event flow.

React Flow