Jump to a problem, path or page
Jump to a problem, path or page
Updates as you edit. Run the review to record it; passing every check completes the problem.
Not yet passed: Client responsibility
Add Client and define its responsibility.
Not yet passed: API service responsibility
Add API service and define its responsibility.
Not yet passed: Database responsibility
Add Database and define its responsibility.
Not yet passed: URL frontier responsibility
Add URL frontier and define its responsibility.
Not yet passed: Crawl worker responsibility
Add Crawl worker and define its responsibility.
Not yet passed: Fetcher responsibility
Add Fetcher and define its responsibility.
Not yet passed: Client → API service
A directed path must connect Client to API service. Arrows represent requests, dependencies, or event flow.
Not yet passed: API service → Database
A directed path must connect API service to Database. Arrows represent requests, dependencies, or event flow.
Not yet passed: API service → URL frontier
A directed path must connect API service to URL frontier. Arrows represent requests, dependencies, or event flow.
Not yet passed: URL frontier → Crawl worker
A directed path must connect URL frontier to Crawl worker. Arrows represent requests, dependencies, or event flow.
Not yet passed: Crawl worker → Fetcher
A directed path must connect Crawl worker to Fetcher. Arrows represent requests, dependencies, or event flow.
Not yet passed: Crawl worker → Database
A directed path must connect Crawl worker to Database. Arrows represent requests, dependencies, or event flow.
Not yet passed: What stops repeated discovery from expanding forever?
Not yet passed: How do several workers respect a host limit?
Not yet passed: What happens after a worker dies with a claimed URL?
Not yet passed: API and data model notes
Write at least 80 characters describing concrete operations, inputs, outputs, state and errors. This check measures completeness, not semantic correctness.
Not yet passed: Trade-offs and failure cases
Write at least 80 characters about alternatives, failure handling and edge cases. Review the reasoning yourself using the reference discussion.
Rule-based educational feedback. Notes are checked by length only; topology checks confirm directed reachability. These checks do not validate semantics, latency, reliability or every valid design.
Use a durable deduplicated URL frontier with expiring claims, shared per-host scheduling and a bounded fetch/parse pipeline. Keep crawl scope and resource policies explicit for every request. Validate approved seeds, scope, robots handling, depth and page budgets. Normalize URL identity without applying unsafe assumptions to query semantics. Persist each discovered URL under a crawl-scoped unique key. Schedule by host with shared concurrency/rate limits, and lease due URLs to workers. Before each fetch, enforce scope, destination restrictions and robots policy. Use bounded redirects, deadlines and compressed/decompressed body limits; every redirect target repeats admission checks. Persist fetch outcome and discovered-link enqueue intents durably before acknowledging completion. Retry transient failures with bounded backoff. Expired leases are recoverable; parser output cannot exceed the remaining crawl budget.
There is no single correct diagram. Explain where yours differs and which requirement justifies it.