⌘

Web Crawler

All changes saved Not started

Choose the policy you would defend in a design review. Choices save as you go.

1.What stops repeated discovery from expanding forever?

Only storing page titlesA canonical URL identity plus crawl depth/page budgetsFollowing every discovered link indefinitely

2.How do several workers respect a host limit?

Each keeps an unrelated local timerIgnore host boundariesAcquire a shared host concurrency/rate permit before fetching

3.What happens after a worker dies with a claimed URL?

Its lease expires and work can be retried idempotentlyThe entire crawl restartsThe URL is considered successfully fetched

Trade-offs, failure cases and tests

Explain a rejected alternative, a failure mode and how you would test the design. At least 80 characters for the completeness check.

0 characters