Load Balancing

A single server has a ceiling. It is built from CPU, memory, file descriptors, and the number of connections the operating system will hold open at once, and when traffic reaches that ceiling the server does not politely slow down. Latency climbs, queues back up, timeouts start firing, and clients retry — which adds load to a machine that is already past its limit. A launch, a sale, or one popular link can find the ceiling in minutes.

Buying a bigger machine buys time. What it cannot buy is survival of that machine’s failure, and there is always a size past which nothing bigger exists. The other direction is to run several ordinary servers and put something in front of them that decides, for every incoming request, which one answers it. That something is a load balancer: the host at the door of a busy restaurant, sending each party to a table so no single waiter ends up with the whole room.

Round-robin and its blind spot

The simplest strategy is to take turns. Keep one integer, send request 1 to the first server, request 2 to the second, request 3 to the third, request 4 back to the first. That is round-robin, and its appeal is real: it holds no opinion about the servers, measures nothing, and therefore has nothing that can go stale or be measured wrong. Over a few thousand requests each server receives the same number of them.

The same number of requests is not the same amount of work. A session check costs a millisecond; generating a monthly report costs thirty seconds of CPU and a lot of memory. Round-robin cannot tell those apart, because it never looks inside a request and never asks a server how it is doing. In the scene below, every fourth request is an expensive report — watch the rotation hand it out right on schedule. The counters climb evenly, and nothing in them says which server is thirty seconds deep in a report.

The even split in that lap is luck. Change the pattern so that every third request is a report — a dashboard that loads two cheap widgets and then one report, over and over — and every report lands on S3, because the pattern now repeats exactly as often as the rotation does. Each round of three requests hands S3 thirty seconds of CPU and S1 and S2 a millisecond each, and the three counters still climb in perfect step.

Servers can differ too, especially in a pool that grew over time. Weighted round-robin fixes that much: give each server a share of the rotation proportional to its capacity, so the older machine takes one request for every three the newer one takes. It is still a fixed opinion decided in advance, not a response to what is happening right now.

Counting what is in flight

To react to reality, the balancer needs a signal that changes with load. The cheapest good one is already in its own hands: how many requests it has open to each server right now. Least-connections sends the next request to whichever server has the fewest in flight.

That small change fixes the report problem. A server tied up on a slow request keeps that connection open, so its in-flight count stays high and new requests stop arriving until it catches up. Nobody had to classify requests as cheap or expensive in advance, and no server had to report its own health honestly.

Both strategies fit behind one interface, which is how real balancers let you swap the policy without touching anything else:

interface Server {
  id: string;
  healthy: boolean;
  inFlight: number;
}

type Strategy = (pool: Server[]) => Server;

function roundRobin(): Strategy {
  let next = 0;
  return (pool) => {
    const server = pool[next % pool.length]!;
    next += 1;
    return server;
  };
}

const leastConnections: Strategy = (pool) =>
  pool.reduce((best, server) => (server.inFlight < best.inFlight ? server : best));

function pickServer(pool: Server[], strategy: Strategy): Server | undefined {
  const healthy = pool.filter((server) => server.healthy);
  return healthy.length === 0 ? undefined : strategy(healthy);
}

Connection count is a proxy, not a measurement of load, and it can mislead. Idle keep-alive connections, WebSockets, and long-polling all inflate the number without costing the server anything, which is why balancers handling those use outstanding requests rather than open connections, and why latency-aware policies — routing by a rolling average of response time — exist for pools where neither count tells the truth.

async function route<T>(
  pool: Server[],
  strategy: Strategy,
  send: (server: Server) => Promise<T>,
): Promise<T> {
  const server = pickServer(pool, strategy);
  if (!server) throw new Error('no healthy server to send to');

  server.inFlight += 1;
  try {
    return await send(server);
  } finally {
    server.inFlight -= 1;
  }
}

const pool: Server[] = ['S1', 'S2', 'S3'].map((id) => ({
  id,
  healthy: true,
  inFlight: 0,
}));
const sentTo: string[] = [];
const send = (ms: number) => async (server: Server): Promise<void> => {
  sentTo.push(server.id);
  await new Promise((resolve) => setTimeout(resolve, ms));
};

const report = route(pool, leastConnections, send(100)); // still running below
for (let i = 0; i < 3; i += 1) await route(pool, leastConnections, send(1));
await report;
console.log(sentTo); // [ 'S1', 'S2', 'S2', 'S2' ]

Health checks

Notice the filter in pickServer. A dead server left in rotation is worse than a smaller pool: round-robin will keep handing it every third request and every one of them will fail. The balancer has to know, without being told, which servers are currently able to answer.

It finds out by probing. Every few seconds it sends a request to a known endpoint on each server. A set number of consecutive failures takes that server out of rotation; probing continues, and a set number of consecutive successes puts it back. Requiring several in a row in both directions is deliberate — it stops one unlucky timeout from ejecting a healthy server, and stops a half-recovered one from flapping in and out.

What the health endpoint checks is a genuine design decision. Returning 200 whenever the process is alive will happily keep a server in rotation whose database pool is exhausted and which fails every real request. Checking every dependency deeply has the opposite failure: when one shared dependency slows down, every server reports unhealthy at the same moment and the balancer empties the entire pool. The endpoint should be cheap, and should test the things that predict this particular server’s ability to serve.

What the balancer cannot fix

All of this assumes the servers are interchangeable. The moment one of them keeps something in local memory that a later request needs — a session, a half-finished upload, an in-process cache of the user’s cart — requests have to return to that same machine.

Sticky sessions do exactly that, usually with a cookie that names each client’s server. They work, and the bill comes later: your distribution is now only as even as the mapping, adding servers does not rebalance existing sessions, and when a server dies its users lose their state, which is the failure you added servers to avoid. The durable fix is to move that state out of the servers into a shared store so that any of them can answer any request. A load balancer distributes requests; it cannot distribute state.