How would you design a rate limiter? | Interview Question

Q: How would you design a rate limiter?

**Target Audience:** Developers preparing for system design interviews. You know what an API and a load balancer are; the algorithms are covered on sibling pages and only summarised here. **Difficulty:** Hard **Verified with:** Node 22.22, Python 3.12, Go 1.24 and Java 21 (all four print identical output). Memory figures were measured on Redis 7.0.15. > **How to read this doc:** Every idea is explained in plain language first, then tagged with `📌 Interview term:` so you also learn the exact words an interviewer expects. All behaviour shown comes from running the example; the **Verified** section quotes the full output. Capacity numbers are worked from stated assumptions. ## 1. Why This Even Matters — A Story First A theme park sells "10 rides per hour" wristbands. If one gate checks wristbands, the rule is easy. But the park has 20 gates, and a visitor can use any of them. If each gate keeps its own tally, a visitor can ride 10 times per gate. If every gate phones the central offic

The problem

Question presented to candidate: "Design the rate limiter for our public API. Requests arrive at a fleet of gateway servers; customers have plans with different limits per API key, and we also want per-IP limits against abuse. Walk me through where it runs, what state it keeps, how the gateways share that state, and what happens when the limiter itself fails."

What a strong answer should cover:

Requirements first: what is limited (API key, user, IP, endpoint), the rules and their source, accuracy needed, latency budget (a limiter check must cost about a millisecond), availability
Placement: a middleware in the API gateway (or edge), keyed per client and rule, returning 429 with Retry-After and remaining-budget headers; client-side and per-service limits as extra layers
Algorithm choice with a reason (usually token bucket, sliding log for small exact limits) and a rules store pushed to gateways
Shared state: local-only limits are wrong when traffic is uneven across nodes; a central store (Redis, sharded by key, atomic Lua) is exact but costs a round trip per request; local batching / token leases cut store calls at some accuracy cost
Failure policy: fail open vs fail closed, timeouts, a local fallback limit; multi-region choices
Capacity numbers (requests per second, keys, memory, Redis shards) and what breaks first at 10x

Clarifying questions expected:

Is the limit hard (billing, abuse) or soft (fairness), and how much over-admission is acceptable?
How many requests per second at peak, how many distinct keys, and how many regions?
Should a limiter outage allow or reject traffic?

Code / implementation expected: Optional — high-level design with numbers; the example simulates gateways sharing a token bucket and the limiter store failing.

rate-limiting

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

Target Audience: Developers preparing for system design interviews. You know what an API and a load balancer are; the algorithms are covered on sibling pages and only summarised here. Difficulty: Hard Verified with: Node 22.22, Python 3.12, Go 1.24 and Java 21 (all four print identical output). Memory figures were measured on Redis 7.0.15.

How to read this doc: Every idea is explained in plain language first, then tagged with 📌 Interview term: so you also learn the exact words an interviewer expects. All behaviour shown comes from running the example; the Verified section quotes the full output. Capacity numbers are worked from stated assumptions.

1. Why This Even Matters — A Story First

A theme park sells "10 rides per hour" wristbands. If one gate checks wristbands, the rule is easy. But the park has 20 gates, and a visitor can use any of them. If each ga

The solution is hidden so you can try first

16 min read · 17 sections

Read the code

Distributed limiter: local vs central vs leased tokens, and fail-open vs fail-closed during a store outage

Open in playground

// A distributed rate limiter in miniature: 4 gateway nodes, one API key limited to
// 100 requests/s (token bucket). The "central store" stands for Redis.
// Fake clock in ms; tokens are integers scaled by 1000 so every language agrees.
class Bucket {
  constructor(cap, perSec) { this.cap = cap * 1000; this.rate = perSec; this.units = this.cap; this.last = 0; }
  refill(t) { this.units = Math.min(this.cap, this.units + (t - this.last) * this.rate); this.last = t; }
  take(t, n) { this.refill(t); const got = Math.min(n, Math.floor(this.units / 1000)); this.units -= got * 1000; return got; }
}

const NODES = 4;
// a burst of 120 requests at t=0; the load balancer is uneven: per 20 requests, 10/6/3/1
const PATTERN = [0, 1, 0, 2, 0, 1, 0, 0, 1, 0, 2, 0, 1, 0, 3, 0, 1, 0, 2, 1];
const burst = [];
for (let i = 0; i < 120; i++) burst.push(PATTERN[i % 20]);

function report(name, admitted, calls, extra) {
  console.log("  " + name.padEnd(30) + "admitted " + String(admitted).padStart(3) + " of 120   store calls " + String(calls).padStart(3) + extra);
}

console.log("== 1. burst of 120 requests at t=0, limit 100");
{
  const per = [0, 0, 0, 0];
  for (const n of burst) per[n]++;
  console.log("  traffic per node: " + per.join(" "));
}
for (const capPerNode of [25, 100]) {
  const local = [];
  for (let i = 0; i < NODES; i++) local.push(new Bucket(capPerNode, capPerNode));
  let ok = 0;
  for (const n of burst) ok += local[n].take(0, 1);
  report("local only, " + capPerNode + " per node", ok, 0, "");
}
{
  const central = new Bucket(100, 100);
  let ok = 0, calls = 0;
  for (const n of burst) { calls++; ok += central.take(0, 1); }
  report("central store, every request", ok, calls, "");
}
{
  const central = new Bucket(100, 100);
  const held = [0, 0, 0, 0];
  let ok = 0, calls = 0;
  for (const n of burst) {
    if (held[n] === 0) { calls++; held[n] += central.take(0, 10); }
    if (held[n] > 0) { held[n]--; ok++; }
  }
  report("central store, leases of 10", ok, calls, "   tokens stranded on nodes " + held.join(" "));
}

console.log("== 2. 250 requests/s for 3 s, round-robin; limit 100/s (burst 20); store down 1000-1999 ms");
for (const policy of ["fail-closed", "fail-open", "fallback to local 25/node"]) {
  const central = new Bucket(20, 100);
  const local = [];
  for (let i = 0; i < NODES; i++) local.push(new Bucket(5, 25));
  const perSec = [0, 0, 0];
  for (let i = 0; i < 750; i++) {
    const t = i * 4, n = i % NODES;
    let ok;
    if (t >= 1000 && t < 2000) {
      ok = policy === "fail-closed" ? 0 : policy === "fail-open" ? 1 : local[n].take(t, 1);
    } else ok = central.take(t, 1);
    perSec[Math.floor(t / 1000)] += ok;
  }
  console.log("  " + policy.padEnd(26) + "admitted per second " + perSec.map((x) => String(x).padStart(3)).join(" "));
}

Discussion

Other approaches, follow-ups and how it went in real interviews.

Sign in to join the discussion.

No comments yet. Share how you would answer it.

The solution is hidden so you can try first

Keep your momentum going.