Web Crawler

Use a durable deduplicated URL frontier with expiring claims, shared per-host scheduling and a bounded fetch/parse pipeline. Keep crawl scope and resource policies explicit for every request.

1. Validate approved seeds, scope, robots handling, depth and page budgets. Normalize URL identity without applying unsafe assumptions to query semantics.
2. Persist each discovered URL under a crawl-scoped unique key. Schedule by host with shared concurrency/rate limits, and lease due URLs to workers.
3. Before each fetch, enforce scope, destination restrictions and robots policy. Use bounded redirects, deadlines and compressed/decompressed body limits; every redirect target repeats admission checks.
4. Persist fetch outcome and discovered-link enqueue intents durably before acknowledging completion. Retry transient failures with bounded backoff. Expired leases are recoverable; parser output cannot exceed the remaining crawl budget.

• Cyclic links enqueue one canonical URL per crawl.
• Several workers respect one shared host budget.
• Redirects outside approved scope or to disallowed addresses stop.
• A crash, compression bomb or unbounded link page cannot lose work or exceed resource budgets.

Trade-offs to explain

• URL deduplication is not content deduplication; multiple URLs may still represent identical pages.
• Politeness and fairness limit throughput intentionally. A crawl budget is necessary even for one approved host.
• Robots directives and site-specific policies need an explicit parser/version and refresh policy; they are not inferred from successful HTTP responses.