Idempotency

A customer taps Pay as their train enters a tunnel. The request gets out: the server charges the card 49.99, records the payment, and sends back 201 Created. The reply never arrives. Ten seconds later the app stops waiting and reports a timeout.

Now the app has to decide what to do, and it has nothing to decide with. A timeout looks the same whether the request was lost on the way out, the reply was lost on the way back, or the server is still working. Give up, and the customer may have paid for an order the screen says failed. Send it again, and the customer may pay twice. The way out is to make sending it again harmless. An operation is idempotent when doing it twice leaves the system in the same state as doing it once.

Why every layer retries

Retrying is the only sensible response to a timeout, and it happens at every layer, often without your code asking. Some client libraries retry dropped connections on their own, a load balancer may resend a request to another server, an app that queued requests while offline replays them, and a message broker redelivers anything not acknowledged. A server that is not safe to retry gets retried anyway.

Some operations are safe by construction. Run SET balance = 90 twice on an account holding 100 and it holds 90; run SET balance = balance - 10 twice and it holds 80, and the customer is ten short. HTTP defines PUT and DELETE as idempotent and POST as not, but a method only states an intent, and some work cannot be phrased as setting a value: creating a payment, sending an email, booking a seat. That work has to be made idempotent, and the standard tool is a key.

A key, and the reply stored under it

The client makes up a unique string, usually a random UUID, when the customer presses Pay, and sends it with the request, typically in an Idempotency-Key header. Every retry of that payment carries the same string. The key names the operation, not the attempt: a key generated inside the retry loop is a new key on every attempt, which is the same as having none.

The server keeps a table of keys and looks the key up before it does anything. A key it has never seen means new work: do it, and store the reply under the key. A key whose work is done gets the stored reply back, and nothing else happens.

Storing the reply, rather than a flag that says seen, matters more than it looks. The retry is the client’s only chance to learn how the payment went. A server that answers a duplicate with an error has prevented the second charge and left the client not knowing the payment id, or whether there was a payment at all.

So a key’s record holds the key, scoped to the account that sent it so that two customers can never collide; a fingerprint of the request body; a state, in progress or done; the stored status and body; and when it was created. The fingerprint catches a bug that would otherwise pass silently: the same key sent with a different request, such as a cart edited to 59.99 after the 49.99 attempt timed out. Replaying the 49.99 reply would be wrong and charging 59.99 would break the key’s promise, so the honest answer is an error — 422 in the code below.

In the scene, each lane loses its first reply and retries once. A server’s badge counts the requests it received; a charges badge counts the times a card was charged.

Both servers received two requests and lost a reply. The lower one differs only in having written down what it did, and in checking before doing it again: the request still arrives twice, and the work happens once.

Two retries at once

The scene retries after the first attempt has finished. A client with a two-second timeout, talking to a server that takes three seconds to charge a card, retries while the first attempt is still running, and now two requests carrying k-7 are in flight at once.

Suppose the server looks the key up, finds nothing, charges, and only then records the key. Both requests look before either records, so both charge. This is check-then-act, the classic race, and it runs straight past the table that was meant to stop it. The fix is to claim the key before the work, in one step that fails if the key already exists — in a database, a unique constraint on the key and an insert that reports whether it won, such as PostgreSQL’s INSERT … ON CONFLICT DO NOTHING. One request wins. The other finds the key in progress and gets 409 Conflict, and its next retry, once the first has finished, gets the stored reply. Write the payment row and the key’s reply in one transaction, too, so that neither can exist without the other.

Expiry, and queues

Keys cannot be kept forever, so they expire, and the window has to outlast every retry that could still arrive: a retry that lands after its key has gone looks exactly like a new payment. Work it out from the retry policy. With a 10-second timeout and waits of 1, 2, 4 and 8 seconds between four retries, the last retry leaves 4 × 10 + 15 = 55 seconds after the first attempt, so minutes would do. An app that queues payments offline and replays them tomorrow needs days. Size the window for the slowest path a retry can take, not the usual one.

Queues are the other source of duplicates, and there they are routine: at-least-once delivery hands a consumer the same message again whenever a worker dies before acknowledging it. A consumer is a server receiving retries, and the message id is its key. A consumer that credits an account and then records the id as a second step can crash between the two, and the redelivery credits it again. Write both in one transaction and the redelivery either finds the id and does nothing, or finds no id because nothing happened.

That only works when the effect lives in the same database as the record. A card charge or an email happens in someone else’s system, and no transaction spans both. There, pass the key along: many payment providers accept an idempotency key of their own, so the system that does the work is also the one that refuses to do it twice.

The lookup, the fingerprint, the claim before the work and the stored reply fit in one function:

type Reply = { status: number; body: unknown };
type KeyRecord =
  | { state: 'in-progress'; fingerprint: string }
  | { state: 'done'; fingerprint: string; reply: Reply };

const keys = new Map<string, KeyRecord>();

async function withIdempotency(
  key: string,
  request: unknown,
  handle: () => Promise<Reply>,
): Promise<Reply> {
  const fingerprint = JSON.stringify(request);
  const seen = keys.get(key);
  if (seen && seen.fingerprint !== fingerprint) {
    return { status: 422, body: 'this key was sent with a different request' };
  }
  if (seen?.state === 'done') return seen.reply;
  if (seen) return { status: 409, body: 'the first attempt is still running' };

  keys.set(key, { state: 'in-progress', fingerprint });
  const reply = await handle();
  keys.set(key, { state: 'done', fingerprint, reply });
  return reply;
}

There is no await between keys.get and the first keys.set, which is what makes the claim atomic here: a Node process runs one request’s synchronous code to completion before another’s can start, so two retries cannot both find an empty slot. Across several servers the Map becomes a table and the claim becomes the insert above. JSON.stringify stands in for a hash of the body, and a real record would also carry its creation time.

async function withIdempotency(
  key: string,
  request: unknown,
  handle: () => Promise<Reply>,
): Promise<Reply> {
  const fingerprint = JSON.stringify(request);
  const seen = keys.get(key);
  if (seen && seen.fingerprint !== fingerprint) {
    return { status: 422, body: 'this key was sent with a different request' };
  }
  if (seen?.state === 'done') return seen.reply;
  if (seen) return { status: 409, body: 'the first attempt is still running' };

  keys.set(key, { state: 'in-progress', fingerprint });
  try {
    const reply = await handle();
    keys.set(key, { state: 'done', fingerprint, reply });
    return reply;
  } catch (error) {
    keys.delete(key); // nothing was stored, so a retry may run the work again
    throw error;
  }
}

let attempts = 0;
const charge = async (): Promise<Reply> => {
  attempts += 1;
  if (attempts === 1) throw new Error('payments database unavailable');
  return { status: 201, body: { paymentId: 'pay_1' } };
};

const payment = { amountCents: 4999 };
await withIdempotency('k-7', payment, charge).catch(() => undefined);
const retry = await withIdempotency('k-7', payment, charge);
console.log(retry.status, attempts); // 201 2 (before the change: 409 1)