Web Crawler

Design a crawler for an explicitly approved set of public websites. URLs may form cycles, pages can be enormous and one host must not consume all worker capacity.

• Enforce crawl scope and an explicit robots/politeness policy.
• Deduplicate normalized URLs and cap crawl depth and total pages.
• Lease frontier work and recover crashed workers.
• Bound network, body and parser resources for untrusted pages.