Design a web crawler from an empty canvas: a URL frontier, a Bloom filter that drops seen URLs, polite fetcher workers, a parser and a page store, taking 1,500 discovered URLs a second.
Unlock it once for 15 credits; everything inside it is free (runs, changes, Submit, results, hints). Sign in with Google to start; new accounts get free credits.
What it must do
Take discovered URLs into a frontier, dropping the ones already seen.
Fetch pages politely: never more than one request at a time to the same site.
Parse each page for its links and text, and store it.
Serve stored pages to the indexer.
Scale
2,000 requests a second: 1,500 discovered URLs pushed to the frontier and 500 stored-page reads by the indexer.
About half the discovered URLs were seen before; the other ~750 a second are about 2 billion new pages a month.
Pages average 100 KB; 10 billion URLs have been seen so far.
Request types to route: enqueue (75%, write), page (25%, read). Budget: $4,000 a month.
Estimate first
New pages fetched a month (billion)
Page store growth a month at 100 KB a page (TB)
Bloom filter memory for 10 billion URLs at a 1% false-positive rate (GB)
Monthly cloud cost of your design (USD)
The test cases
You start from an empty canvas. When you run the tests, each one simulates your design with its own traffic and failures:
Test
What it does
Passes when
Normal day
The normal day: the traffic in the brief, nothing broken.
Success ≥ 99.9%; p99 latency ≤ 100 ms (enqueue); p99 latency ≤ 100 ms (page)
A huge sitemap lands
At 5 s a big site’s sitemap is discovered: traffic triples for 8 s.
Success ≥ 99%; p99 latency ≤ 150 ms (enqueue)
A server dies
At 4 s one server of the first app service on the request path dies for 10 s.
Success ≥ 99%
Cache flushed
At 5 s the cache on the request path is emptied, as after a restart or a bad deploy.
Success ≥ 99%; p99 latency ≤ 500 ms (page)
Budget: $4,000/month
The normal day's design must cost at most $4,000 a month.
Cost ≤ $4,000/month
Must include
A URL frontier. URLs wait in a durable frontier, partitioned by host, until a fetcher takes them: fetching never blocks discovery.
A seen-URL filter. Ten billion URLs do not fit in a set; a Bloom filter answers “seen it?” from a few GB of memory.
A page store. Billions of fetched pages are blobs: they go to object storage.
Fetcher workers. Fetching takes hundreds of milliseconds per page: workers pull from the frontier at their own pace.