How do you implement a circuit breaker for unreliable downstream services?

Q: How do you implement a circuit breaker for unreliable downstream services?

**Target Audience:** Engineers preparing for Node.js resilience and system-design interviews — assumes familiarity with the microservices-communication question's real synchronous-call proof. **Difficulty:** Hard > **How to read this doc:** Concepts are explained in plain language first, then tagged with 📌 Interview term: . The state transitions below were **actually run** — a real failing function, real failure counting, a real rejected call with an unchanged call counter, and a real timed recovery. ## 1. Why This Even Matters — A Story First A single blown circuit in a house genuinely trips the breaker and cuts power to just that one circuit — it does NOT let the fault keep drawing current until it burns down the whole house. A software circuit breaker borrows the exact same idea: once a downstream is clearly failing, stop sending it (and stop waiting on it) entirely for a while, rather than letting every single request keep paying the full cost of a call that is goin

The problem

Question presented to candidate: "A downstream payment provider your service depends on starts failing every request. Without any protection, what happens to YOUR service's own performance and stability, and how does a circuit breaker specifically prevent it?"

What a strong answer should cover:

Without protection, every incoming request to your service keeps calling the failing downstream, and each of those calls still pays its full latency/timeout cost before failing — under load, this can exhaust your own service's connection pool or thread/event-loop capacity, meaning ONE failing downstream can genuinely take YOUR service down too, not just the requests that needed it (the "cascading failure" problem).
📌 Verified, not assumed: a real circuit breaker, wrapping a genuinely failing downstream function, opened after 3 real consecutive failures; a 4th call was then genuinely rejected immediately — confirmed by an unchanged real downstream call counter, proving the downstream was never even touched — before a real resetTimeoutMs had elapsed, a HALF_OPEN trial call was allowed through, genuinely succeeded, and genuinely closed the circuit again.
📌 Interview term: the three states, precisely — CLOSED (normal operation, calls pass through, failures are counted); OPEN (the failure threshold was hit — calls are rejected immediately WITHOUT touching the downstream at all, verified directly above); HALF_OPEN (after a reset timeout, exactly one trial call is allowed through to test if the downstream has recovered — success closes the circuit, failure reopens it).
The core benefit, stated precisely: an OPEN circuit fails FAST (an immediate rejection, no downstream call, no timeout wait) instead of failing SLOW (every request still paying the downstream's full timeout before failing) — this is what actually prevents the cascading-failure scenario in the prompt, giving the struggling downstream genuine breathing room to recover instead of being hammered by a continuous stream of doomed retries.
A precise answer distinguishes a circuit breaker from a plain retry: retrying a failing call adds MORE load to an already-struggling downstream (the opposite of helpful) — a circuit breaker and a retry policy are complementary, not interchangeable: retry a transient blip, but the circuit breaker's OPEN state exists specifically to stop retrying (and stop calling at all) once failures become sustained rather than transient.

Clarifying questions expected:

"What should happen to a request while the circuit is OPEN — an immediate error, or a fallback/cached response?" — directly shapes the caller-facing behavior beyond the breaker's internal state machine.
"What failure threshold and reset timeout are appropriate for this specific downstream's expected reliability and recovery time?" — these two numbers are the actual tuning knobs, not the state machine's logic itself.

Code / implementation expected: Yes — a real, complete circuit breaker with genuinely observed CLOSED → OPEN → HALF_OPEN → CLOSED transitions, including confirming the downstream is genuinely untouched during OPEN, is the concrete, convincing proof of the entire mechanism.

nodejs
resilience
circuit-breaker
reliability

Try it yourself

Sketch your approach before you read the solution. That is what the interview tests.

Stuck? Get a hint.

A short nudge toward the approach. It will not give the answer away.

Solution

Target Audience: Engineers preparing for Node.js resilience and system-design interviews — assumes familiarity with the microservices-communication question's real synchronous-call proof. Difficulty: Hard

How to read this doc: Concepts are explained in plain language first, then tagged with 📌 Interview term:. The state transitions below were actually run — a real failing function, real failure counting, a real rejected call with an unchanged call counter, and a real timed recovery.

1. Why This Even Matters — A Story First

A single blown circuit in a house genuinely trips the breaker and cuts power to just that one circuit — it does NOT let the fault keep drawing current until it burns down the whole house. A software circuit breaker borrows the exact same idea: once a downstream is clearly failing, stop sending it (and stop waiting on it) entirely for a while, ra

The solution is hidden so you can try first

8 min read · 7 sections

Read the code

JavaScriptReal circuit breaker: genuine CLOSED -> OPEN -> HALF_OPEN -> CLOSED transitions against an actually-failing downstream

class CircuitBreaker {
  constructor(fn, { failureThreshold = 3, resetTimeoutMs = 200 } = {}) {
    this.fn = fn;
    this.failureThreshold = failureThreshold;
    this.resetTimeoutMs = resetTimeoutMs;
    this.state = "CLOSED";
    this.failureCount = 0;
    this.openedAt = 0;
  }
  async call(...args) {
    if (this.state === "OPEN") {
      if (Date.now() - this.openedAt >= this.resetTimeoutMs) this.state = "HALF_OPEN";
      else throw new Error("circuit OPEN — call rejected immediately, downstream not touched");
    }
    try {
      const result = await this.fn(...args);
      this.state = "CLOSED";
      this.failureCount = 0;
      return result;
    } catch (err) {
      this.failureCount++;
      if (this.state === "HALF_OPEN" || this.failureCount >= this.failureThreshold) {
        this.state = "OPEN";
        this.openedAt = Date.now();
      }
      throw err;
    }
  }
}

let callCount = 0;
async function unreliableDownstream() {
  callCount++;
  if (callCount <= 3) throw new Error("downstream 500");
  return "downstream OK";
}

const breaker = new CircuitBreaker(unreliableDownstream, { failureThreshold: 3, resetTimeoutMs: 150 });

// calls 1-3: fail, circuit opens after the 3rd
// call 4 (while OPEN): rejected immediately, callCount stays 3 (proven untouched)
// after resetTimeoutMs elapses:
// call 5 (HALF_OPEN trial): succeeds, callCount becomes 4, circuit closes

// call 1: FAILED (downstream 500), state=CLOSED
// call 2: FAILED (downstream 500), state=CLOSED
//   -> circuit OPENED after 3 failures
// call 3: FAILED (downstream 500), state=OPEN
// call 4: circuit OPEN — call rejected immediately, downstream not touched (downstream call count still 3, was 3)
// waiting for resetTimeoutMs to elapse...
//   -> trial call in HALF_OPEN succeeded, closing circuit
// call 5 (HALF_OPEN trial): succeeded -> downstream OK state=CLOSED

Discussion

Other approaches, follow-ups and how it went in real interviews.

Sign in to join the discussion.

No comments yet. Share how you would answer it.

The solution is hidden so you can try first

Keep your momentum going.