Web Crawler System – System Design Interview Preparation

Practice designing a web crawler like Googlebot. Cover URL frontier, politeness, distributed crawling, and deduplication.

This is a full mock interview round, not a written solution to read through. The AI interviewer poses the Web Crawler design problem, pushes back on your choices, and asks the follow-up questions a real interviewer would: how the system behaves as traffic grows, where the bottlenecks move, which consistency guarantees you are trading away, and what happens when a component fails. You get structured feedback and a score on requirement gathering, high level architecture, data modelling, scalability and communication at the end of every attempt.

Designing Web Crawler is a recurring question in High Level Design (HLD) rounds at Google, Amazon, Meta, Microsoft, Netflix and Uber, and at high-growth product companies hiring SDE-2 and above. Working through it out loud - repeatedly, under time pressure - is what separates candidates who know the components from candidates who can defend a design.

What this question is really testing

Politeness and deduplication at scale - crawling a domain without overwhelming it, and not re-fetching the same content under a million URLs.

Scale to establish before drawing anything

1B pages a month is around 400 pages a second sustained, across millions of domains.
A page averages 100 KB, so a month is tens of terabytes before any processing.
The URL frontier holds billions of entries and must be prioritised, not FIFO.

Core components

URL frontier – Prioritised queues with per-domain sub-queues, which is how politeness is enforced structurally.
Fetcher pool – Distributed workers honouring robots.txt and per-domain rate limits.
Content deduplicator – Hash or simhash of normalised content. Near-duplicate detection is the harder half.
URL deduplicator – Bloom filter over seen URLs, with canonicalisation before the check.
Parser and extractor – Pulls links and content, feeding new URLs back to the frontier.
Robots cache – Per-domain rules, cached so they are not re-fetched constantly.

Trade-offs to defend

Politeness by frontier structure – One sub-queue per domain with a next-fetch time is how you guarantee you never hammer a host. Doing it with a global rate limit does not work.
Bloom filter false positives – A false positive silently skips a page forever. Sizing the filter is a real decision with a real cost.
Breadth-first versus priority – Breadth-first is fair and spends budget on low-value pages. Priority by estimated importance is better and needs a signal.

Follow-up questions to expect

A site has a calendar generating infinite URLs. What stops you?
The same article is reachable at 50 URLs across mirrors. How many times do you store it?
A domain is slow and has 10M pages. How do you avoid it starving every other domain?