Web Crawler · Analysis

01 · Frame the problem

Understand the central challenge

Design a crawler for an explicitly approved set of public websites. URLs may form cycles, pages can be enormous and one host must not consume all worker capacity.

The frontier determines uniqueness and scope, while the fetch pipeline must respect shared host limits and bounded resource use. URL identity, redirect policy and claim recovery need explicit definitions before scaling workers.

02 · Find the design pressure

Start simple, then identify the gap

Begin with a durable deduplicated URL frontier. A per-process visited set forgets progress on restart and cannot coordinate workers. Shared claims and host budgets add parallelism without multiplying fetches or violating the crawl's scope.

Use a durable deduplicated URL frontier with expiring claims, shared per-host scheduling and a bounded fetch/parse pipeline. Keep crawl scope and resource policies explicit for every request. Validate approved seeds, scope, robots handling, depth and page budgets. Normalize URL identity without applying unsafe assumptions to query semantics. Persist each discovered URL under a crawl-scoped unique key. Schedule by host with shared concurrency/rate limits, and lease due URLs to workers. Before each fetch, enforce scope, destination restrictions and robots policy. Use bounded redirects, deadlines and compressed/decompressed body limits; every redirect target repeats admission checks. Persist fetch outcome and discovered-link enqueue intents durably before acknowledging completion. Retry transient failures with bounded backoff. Expired leases are recoverable; parser output cannot exceed the remaining crawl budget.

03 · Define success

Requirements to preserve

Enforce crawl scope and an explicit robots/politeness policy.
Deduplicate normalized URLs and cap crawl depth and total pages.
Lease frontier work and recover crashed workers.
Bound network, body and parser resources for untrusted pages.

Workload: 5,000 reads/s, 2,000 writes/s, 1000 B records, 300 ms read p99 target. These are exercise assumptions; the Solution includes capacity formulas.

04 · Identify the correctness boundary

Consistency, ownership and failure boundaries

Canonical URL identity must follow a documented normalization policy; overly aggressive normalization can merge distinct resources. Claim frontier items with expiring ownership, but enforce shared host budgets across all workers. Redirects, decompression and parsing each need their own size and deadline limits.

05 · Make it concrete

Reason through a worked example

Two pages link to the same canonical URL. One frontier row is created and one worker claims it. If the worker crashes, its claim becomes eligible again. A redirect to a disallowed host is rejected before fetching, even though the original URL was inside the approved crawl scope.