Distributed Rate Limiter – System Design Interview Preparation

Practice designing a distributed rate limiter. Cover token bucket, sliding window algorithms, and graceful degradation.

This is a full mock interview round, not a written solution to read through. The AI interviewer poses the Rate Limiter design problem, pushes back on your choices, and asks the follow-up questions a real interviewer would: how the system behaves as traffic grows, where the bottlenecks move, which consistency guarantees you are trading away, and what happens when a component fails. You get structured feedback and a score on requirement gathering, high level architecture, data modelling, scalability and communication at the end of every attempt.

Designing Rate Limiter is a recurring question in High Level Design (HLD) rounds at Google, Amazon, Meta, Microsoft, Netflix and Uber, and at high-growth product companies hiring SDE-2 and above. Working through it out loud - repeatedly, under time pressure - is what separates candidates who know the components from candidates who can defend a design.

What this question is really testing

Choosing an algorithm whose burst behaviour you can defend, then making counters correct across many servers without a round trip per request.

Scale to establish before drawing anything

1M users, 10 requests a second each at peak - the limiter sits in front of everything, so its own latency budget is about 1ms.
A counter per user per window is roughly 20 bytes; 1M users is under 100 MB, so it fits in memory.
Distributed across 20 gateway nodes, so a naive per-node limit lets through 20x the intended rate.

Core components

Limiter middleware – Runs in the gateway, before auth and routing. Rejects with 429 and a Retry-After header.
Counter store – Redis or similar, with atomic increment-and-expire. The atomicity is the whole point.
Rule engine – Limits per user, per API key, per IP and per endpoint, resolved to the strictest match.
Local cache – Short-lived per-node counts to avoid a network hop on every request, at the cost of slight over-admission.

Trade-offs to defend

Fixed window versus sliding – Fixed window is one counter and allows a 2x burst across the boundary. Sliding log is exact and stores every timestamp. Sliding window counter is the usual compromise, and naming why is the answer.
Token bucket versus leaky bucket – Token bucket permits bursts up to the bucket size; leaky bucket smooths output. Which one you want depends on whether the downstream can absorb a spike.
Centralised versus local counters – Centralised is correct and adds a hop; local is fast and over-admits. Most production limiters accept the over-admission and say so.

Follow-up questions to expect

Redis is unavailable. Do you fail open and let everything through, or fail closed and reject everything? Defend the choice for this system.
A customer is at 99% of their limit on two gateway nodes at once. What does each node see?
Add a per-endpoint limit on top of a per-user one. Which applies, and where is that resolved?