Circuit Breakers

It is the evening of a sale, and the payments provider has started to hang. It does not refuse connections or return errors; it accepts each charge and says nothing until checkout’s HTTP client gives up thirty seconds later. Checkout keeps calling, because nothing in checkout knows better. Within seconds every worker thread is sitting in a call to payments, and then the product pages, the cart and the order history start timing out too, although none of them touch payments, because no thread is left to serve them.

Nothing was wrong with checkout. It was taken down by waiting politely for an answer that was never coming. What it needed was a way to notice that payments was failing and stop asking for a while: to give up in a millisecond instead of thirty seconds, and to check back later. That is a circuit breaker.

How one slow service spreads

A dependency that fails quickly is cheap to live with. The one that hurts is the slow one, because every call to it holds something in the caller for as long as it waits: a thread, a connection from a pool, a socket, the memory for the request. How many are held at once is simple arithmetic: the rate calls arrive at, multiplied by how long each one waits. That is Little’s law.

Put illustrative numbers on it. Checkout handles 50 requests a second, and each one charges a card. Payments normally answers in 80 ms, so about 50 × 0.08 = 4 calls are in flight at any moment, and a pool of 200 threads barely notices them. Now payments hangs and the timeout is 30 seconds. Calls in flight become 50 × 30 = 1,500, and the pool of 200 is full after 200 ÷ 50 = 4 seconds. From then on checkout cannot accept anything, including requests that never go near payments.

Retries make it worse. If each charge is attempted three times before checkout gives up, payments receives 150 calls a second instead of 50, just as it is struggling, and each checkout request holds its thread for three timeouts instead of one. Even with a 2-second timeout, that is 6 seconds a request and 50 × 6 = 300 threads wanted from a pool of 200. A failure in one service has become an outage in its caller, and the caller’s callers are next. That is a cascading failure.

A shorter timeout is necessary but not sufficient. At 2 seconds without retries checkout holds 50 × 2 = 100 threads and survives, but every customer still waits two seconds for an error the previous hundred already got. The evidence that payments is down is in the caller’s own recent history, and nothing uses it.

Closed, open, half-open

A circuit breaker is a small object that wraps every call to one dependency and remembers how those calls have been going. The name comes from the switch in a fuse box that cuts the power when too much current flows. It also explains the detail that trips people up: in a circuit, closed means current flows and open means it does not. An open breaker is the one refusing calls.

There are three states. Closed is normal: every call goes through, and the breaker counts failures, meaning timeouts, refused connections and server errors. A success resets the count. When the count reaches a threshold, say 3 failures in a row, the breaker trips open. While open it does not call the dependency at all, and every call fails immediately, in the time it takes to read a field. That is failing fast, and it does two jobs: callers stop waiting, and a struggling dependency stops receiving traffic it cannot handle, which is often what it needs to recover.

An open breaker cannot tell when the dependency is healthy again, because it has stopped asking. So after a cool-down, say 10 seconds, it moves to half-open and lets exactly one call through as a trial. If the trial succeeds, the breaker closes and traffic resumes; if it fails, the breaker opens again for another full cool-down. Every other call that arrives while the trial is out still fails fast. One request decides, so a dependency that is still down receives one call per cool-down rather than a flood. A load balancer’s health check asks the same question with a probe on a timer; a breaker asks with real calls, from inside the caller.

The payments badge counts the calls that reached it and the fallback’s counts the calls answered without it, so the first stops climbing while the breaker is rust. The scene sends one charge at a time. At 50 a second, every charge sent before the third timeout fires also waits its 2 seconds, about a hundred of them: a breaker limits the damage from the moment it trips, not before.

The whole state machine fits in one class. The clock is a field, so a test can move time forward instead of waiting for it:

class CircuitBreaker {
  state: 'closed' | 'open' | 'half-open' = 'closed';
  threshold = 3; // failures in a row that trip it open
  coolDownMs = 10_000; // how long it stays open before one trial call
  now = Date.now; // the clock, replaceable so a test need not wait
  private failures = 0;
  private openUntil = 0;

  async call<T>(request: () => Promise<T>): Promise<T> {
    if (this.state === 'open' && this.now() >= this.openUntil) {
      this.state = 'half-open'; // this call is the trial; others fail fast
    } else if (this.state !== 'closed') {
      throw new Error('circuit open: failing fast');
    }
    try {
      const result = await request();
      this.state = 'closed';
      this.failures = 0;
      return result;
    } catch (error) {
      this.failures += 1;
      if (this.state === 'half-open' || this.failures >= this.threshold) {
        this.state = 'open';
        this.openUntil = this.now() + this.coolDownMs;
      }
      throw error;
    }
  }
}

Notice what call cannot see. It learns about a failure only when request rejects, and a request that hangs never rejects, so around a client with no timeout this breaker would stay closed through the whole outage in the opening. The timeout is not the breaker’s job, but the breaker is blind without one. Checkout calls breaker.call(() => payments.charge(order)) and catches the error to choose what to return instead.

function withTimeout<T>(
  request: () => Promise<T>,
  ms: number,
): () => Promise<T> {
  return () =>
    new Promise<T>((resolve, reject) => {
      const timer = setTimeout(
        () => reject(new Error(`no answer after ${ms} ms`)),
        ms,
      );
      request()
        .then(resolve, reject)
        .finally(() => clearTimeout(timer));
    });
}

const breaker = new CircuitBreaker();
const hang = (): Promise<string> => new Promise(() => {}); // never settles

for (let i = 0; i < 3; i += 1) {
  await breaker.call(withTimeout(hang, 50)).catch((error: Error) => {
    console.log(error.message); // no answer after 50 ms (three times)
  });
}
console.log(breaker.state); // open

await breaker.call(hang).catch((error: Error) => console.log(error.message));
// circuit open: failing fast — at once, with no timeout needed

What to return instead

Failing fast only helps if the caller has something better to return, and choosing it is a product decision. There are three honest kinds of answer. The first degrades the feature: accept the order, mark the payment pending and settle it from a queue once payments is back, provided the business can live with the occasional card that declines an hour later. The second is a cached value, such as the last known exchange rate, good enough for a page that is checked again before money moves. The third is a clear error, returned quickly. Where no substitute is safe, because you cannot pretend a card was charged, a fast and honest error still beats the same error thirty seconds later.

Keep the fallback cheap and away from the failing dependency. It runs on every call while the breaker is open, which is exactly when the system is most stressed.

Choosing the numbers, and the limits

Three numbers need choosing, and each can be wrong in both directions. The timeout should come from measured latency, not from a client library’s default: a little above the slowest answer you still consider healthy, such as the 99th percentile, and short enough that rate × timeout fits comfortably in the pool. The threshold should not trip on noise. Consecutive failures is the simplest rule; many implementations use a failure rate over a sliding window instead, with a minimum number of calls, so that two failures out of three at 4 a.m. do not read as a 67% failure rate. The cool-down trades two harms: too short and trials keep hitting a service that is still down; too long and a healthy one is refused for no reason.

Four limits are worth knowing. Keep one breaker per dependency: a single breaker around everything lets one failing service shut off all the healthy ones. Its state is per instance: each process keeps its own count, so twenty instances of checkout each discover the outage for themselves and each send their own trials. It is not a retry policy: a breaker never retries, it refuses. Retry, if at all, outside it and with a growing delay, so that each attempt is counted and an open breaker refuses the retries too. And it can flap: a dependency that fails about half its calls can trip, pass one lucky trial, close, and trip again. A rate-based threshold, and a cool-down that grows after each failed trial, both damp it.