Caching

Every visitor to your product page triggers the same query: fetch the category tree, join it to the product counts, sort the result. The database walks the same rows and returns the same answer it returned to the previous visitor a few milliseconds ago, and to the thousand visitors before that. Nothing about the answer has changed since yesterday. You are paying for the same computation over and over because nobody kept the result.

The fastest work is the work you do not repeat. A cache is a place to keep an answer you already produced, close to whoever needs it next — a note stuck to your monitor with the number you keep looking up, instead of walking to the filing cabinet each time. The note is instant. It is also, from the moment the number changes, quietly wrong, and that tension is the whole subject.

Cache-aside

The most common pattern puts the application in charge. Before doing the expensive work, look in the cache. If the value is there and still valid — a hit — return it and stop. If it is not — a miss — do the real work, store the result in the cache, and return it. The next request for that key is a hit.

It is called cache-aside because the cache sits beside the source of truth rather than in front of it, and the application decides what goes in. Two consequences follow. Only requested data is ever cached, so the cache fills itself with whatever is actually popular. And if the cache disappears completely, every request becomes a miss and the system still returns correct answers, just slowly — a failure mode you can survive at 3am.

In the scene below, the badge on each box counts arrivals. Watch the two badges diverge: the cache keeps taking requests while the database stops being asked.

Hit rate, and what moves it

A cache earns its keep through one number: the fraction of requests it answers without the underlying work. At a high hit rate the database sees a trickle of traffic instead of a flood. At a low hit rate the system has taken on a network hop, a serialization step, a pile of memory, and a second copy of the data to reason about, in exchange for nothing. A cache that misses is not neutral; it is overhead.

Put numbers on it, because the intuition here is usually wrong. At 1,000 reads a second, a 90% hit rate sends 100 reads to the database, and a 99% hit rate sends 10. Those two hit rates look nine points apart; to the database they are a tenfold difference in load. What the database feels is the miss rate, so the last few points of hit rate are worth far more than the first fifty. The lap in the scene above, with three keys and eight reads, hits 5 in 8. Real traffic, with the same few keys asked for thousands of times, does much better.

What moves that number is how concentrated the traffic is. Real access patterns are lopsided — a small set of keys accounts for most requests — and caches exploit exactly that skew. When the cache fills, an eviction policy decides what to drop, usually least-recently-used: the entry nobody has touched for the longest is the cheapest guess to throw away.

The other lever is time to live, or TTL: every entry gets an expiry, after which it is treated as absent and the next request repopulates it. TTL is crude — it expires entries that were still perfectly correct — but it is the single most valuable safety valve in caching, because it puts a hard ceiling on how wrong you can be. A stale price with a 30-second TTL is a 30-second problem. Set it per key, not globally: a currency rate and a country list do not deserve the same number.

Invalidation, the part that is actually hard

Storing a value is easy. Knowing when the stored value stopped being true is not, and this is where caching stops being free.

There are two ways to get it wrong. Serve stale data: a write lands in the database, the cache still holds the previous copy, and users see an old answer with total confidence. Or invalidate too eagerly: clear so aggressively that the cache is never warm, and the complexity stays while the benefit goes.

TTL bounds staleness by the clock. Explicit invalidation — deleting or rewriting cache entries when a write happens — bounds it by event, which is tighter but demands something harder: knowing every key a given write invalidates. Rename one user and you may need to touch their profile payload, the comment list that embeds their name, the team roster, and the search document. Miss one and you have a stale corner that nobody notices for weeks. This is why experienced teams cache aggressively at the edges and cautiously in the middle, and why TTLs stay short on anything that changes.

One more failure mode deserves naming. When a popular key expires, every request that arrives in the next instant misses simultaneously, and all of them go to the database at once — a stampede, and it lands precisely when you are busiest. The standard defenses are to let a single request perform the refresh while the others wait or receive the old value, and to add random jitter to TTLs so that keys written together do not all expire together.

Cache-aside with a TTL is short enough to read in one sitting:

interface User {
  id: string;
  name: string;
}

interface CacheEntry {
  value: User;
  expiresAt: number;
}

const TTL_MS = 60_000;
const cache = new Map<string, CacheEntry>();

async function getUser(
  id: string,
  loadUser: (id: string) => Promise<User>,
): Promise<User> {
  const cached = cache.get(id);
  if (cached && cached.expiresAt > Date.now()) return cached.value;

  const user = await loadUser(id);
  cache.set(id, { value: user, expiresAt: Date.now() + TTL_MS });
  return user;
}

The stampede is visible in that code: nothing prevents two concurrent misses for the same id from both calling loadUser. A single in-flight map — key to pending promise — collapses them into one database read, and is usually the first thing added to a cache like this.

const inFlight = new Map<string, Promise<User>>();

async function getUser(
  id: string,
  loadUser: (id: string) => Promise<User>,
): Promise<User> {
  const cached = cache.get(id);
  if (cached && cached.expiresAt > Date.now()) return cached.value;

  const pending = inFlight.get(id);
  if (pending) return pending;

  const load = loadUser(id)
    .then((user) => {
      cache.set(id, { value: user, expiresAt: Date.now() + TTL_MS });
      return user;
    })
    .finally(() => inFlight.delete(id));
  inFlight.set(id, load);
  return load;
}

let calls = 0;
const load = async (id: string): Promise<User> => {
  calls += 1;
  return { id, name: 'Ada' };
};

await Promise.all([getUser('42', load), getUser('42', load)]);
console.log(calls); // 1 (it was 2 before the change)