Message Queues

Checkout does six things when an order is placed. It charges the card, writes the order, emails a receipt, reserves stock in the warehouse system, pushes an event to analytics, and pings the fraud service. Five of those are fast. The warehouse system is somebody else’s API, and on the afternoon of a sale it slows to a crawl — so checkout, which is waiting on it, slows to a crawl too. Customers who only wanted to buy something watch a spinner while a warehouse in another company thinks about stock levels.

Nothing there is wrong except the waiting. The card was charged, the order is real, and the warehouse will hear about it whichever second it hears. What the code does not have is a way to say later — a place to put the work down so the customer’s request can finish now. That place is a queue, and having one changes the shape of the whole system.

Handing off instead of waiting

Picture the ticket rail in a busy kitchen. A server takes an order, clips the ticket to the rail, and is back on the floor in two seconds. The kitchen works the rail at whatever pace the kitchen can manage. Nobody stands at the pass holding a notepad until a plate appears, and when forty people arrive at once the rail simply gets longer.

A message queue is that rail. The producer — checkout — writes a small message saying order 1041 was placed and returns. A separate process, a consumer or worker, picks the message up later and does the slow part. The two never call each other and never wait for each other; they only ever agree on the shape of the message. That is what decoupling means here, and the important half of it is decoupling in time: the producer and the consumer no longer have to be fast at the same moment, or even be running at the same moment.

That is what buys you the burst. Traffic is not smooth — it arrives in spikes that can run well above the average for minutes at a time. Sizing every synchronous component for the spike is expensive and mostly idle. With a queue in the middle you size the producers for the spike, because accepting a message is cheap, and size the consumers for something closer to the average. The spike turns into a backlog, and a backlog drains.

Depth is the number to watch

The queue’s honest health signal is its depth: how many messages are waiting right now, which is arrivals minus completions. A depth that rises during a spike and falls afterwards is a queue doing its job. A depth that rises and never comes back down means the consumers cannot keep up with the average, not just the peak, and no amount of buffer will save that — it only delays the moment you notice.

Put illustrative numbers on it. Two workers that each finish 60 orders a minute clear 120 a minute between them. A sale brings 300 a minute for ten minutes, so the queue grows by 180 a minute and ends the sale 1,800 deep, while every checkout still returned at once. When traffic falls back to 60 a minute, the workers gain 60 a minute on the backlog and clear it in half an hour; the last order of the sale has its stock reserved fifteen minutes late, not never. If everyday traffic were 150 a minute instead, the queue would grow by 30 a minute with no sale at all, and that is the floor that keeps rising.

The fix for a rising floor is more consumers, which a queue makes almost free: they all pull from the same place, so starting a third and a fourth worker needs no configuration anywhere else. This is also why the work has to be genuinely independent per message. Two workers pull in arrival order, but they finish in whatever order their messages take, so anything that depends on strict global ordering needs more than a plain queue.

Read the badges carefully, because they are not depth. The queue’s badge counts every message that has arrived — including one that came back — and a worker’s counts the messages delivered to it. Depth is the difference: subtract the two worker badges from the queue’s. Step to the end of the run and it is 6 − 3 − 3 = 0, an empty queue, which is exactly where a healthy one returns to.

At-least-once, and the duplicate it hands you

A broker cannot know a worker succeeded until the worker says so. So delivery is a loan: the message goes out, the worker does the job, and the worker sends an acknowledgement — an ack — after which the broker drops the message. If the ack never comes, because the worker crashed or the network did, the broker has only one safe option, and it takes it: hand the message to somebody else.

That guarantee is called at-least-once, and it is the one nearly every broker gives you by default. The alternative — acknowledging before doing the work — is at-most-once, and it trades duplicates for lost messages, which is almost never the trade you want. Exactly-once delivery is not on the menu; what systems sell under that name is at-least-once delivery plus deduplication at the consumer.

So build the consumer to survive being handed the same message twice. That property is idempotency, and the usual implementation is a natural key the consumer records: charge against order:1041 rather than “charge this card”, and check whether that key was already done before doing it. Two other habits go with it. Give every message a stable id so deduplication has something to key on. And cap the retries — a message that has failed a handful of times is usually poison, and it belongs in a dead-letter queue where a human can look at it, rather than in a loop that never ends.

A queue and one worker turn fit in a few lines:

interface Job {
  id: string;
  orderId: string;
}

const waiting: Job[] = [];
const done = new Set<string>();

/** The producer's entire involvement: append, and return to the customer. */
function publish(job: Job): void {
  waiting.push(job);
}

/** One consumer turn. Depth is `waiting.length` — arrivals minus completions. */
async function work(run: (job: Job) => Promise<void>): Promise<void> {
  const job = waiting.shift();
  if (!job) return;
  if (done.has(job.id)) return; // the duplicate a redelivery hands you
  try {
    await run(job);
    done.add(job.id); // the ack: only now is the message really gone
  } catch {
    waiting.push(job); // no ack, so it goes back out — at-least-once
  }
}

The catch is the whole guarantee in one line, and the done check is the defense against it, though not a complete one: a worker that crashes after run(job) finishes but before done.add runs never acks, so the job is delivered and run again. Everything a real broker adds — durability across a restart, visibility timeouts so a dead worker’s message is reclaimed rather than lost forever, retry limits, dead-letter queues — is a more careful version of those two lines.

const MAX_ATTEMPTS = 3;
const attempts = new Map<string, number>();
const deadLetters: Job[] = [];

async function work(run: (job: Job) => Promise<void>): Promise<void> {
  const job = waiting.shift();
  if (!job) return;
  if (done.has(job.id)) return;
  try {
    await run(job);
    done.add(job.id);
    attempts.delete(job.id);
  } catch {
    const failures = (attempts.get(job.id) ?? 0) + 1;
    attempts.set(job.id, failures);
    if (failures < MAX_ATTEMPTS) {
      waiting.push(job); // still worth another try
    } else {
      deadLetters.push(job); // poison: parked where a human will look
      attempts.delete(job.id);
    }
  }
}

let calls = 0;
const reserveStock = async (job: Job): Promise<void> => {
  calls += 1;
  if (job.orderId === '1042') throw new Error('order 1042 has no line items');
};

publish({ id: 'm1', orderId: '1041' });
publish({ id: 'm2', orderId: '1042' });
for (let turn = 0; turn < 10; turn += 1) await work(reserveStock);

console.log(calls, waiting.length); // 4 0: one success, three tries at the poison
console.log(deadLetters.map((job) => job.orderId)); // [ '1042' ]