Horizontal vs Vertical Scaling

The dashboard is unambiguous. Every weekday between nine and ten the API’s response times climb, the CPU on the one server that runs it sits pinned near the top, and support gets the same three tickets. Nobody has to argue about what is happening — the machine is out of room. The only question is what to do about it, and there are exactly two answers.

Get a bigger machine, or get another machine. That is the whole taxonomy. Vertical scaling means growing the box you have: more cores, more memory, faster disks. Horizontal scaling means running more boxes and spreading the work across them. They sound like two settings of the same dial, and they are not — one is a purchase and the other is an architecture.

Growing the box

A delivery company with one van and too many parcels can buy a bigger van. It is a good move: the driver, the route, and the paperwork are all unchanged, and by Monday the problem is gone.

Vertical scaling is that van. You resize the instance, restart, and the application is faster with no code changed, no new failure modes, and nothing new to reason about. There is one process holding one copy of the state, so a session in memory stays valid, an in-process cache stays coherent, and a lock is just a lock. This is genuinely underrated: an enormous number of systems will never need anything else, and reaching for a distributed architecture before you have to is how teams buy a year of complexity to solve a Tuesday.

It has two walls, though, and you meet them in this order. The first is the ceiling. Machines come in sizes, the largest size is finite, and past a certain point each step up costs disproportionately more for the same increment of capacity. When you are running the biggest instance your provider rents, the strategy is simply out of moves. The second wall is that one machine is one machine: while it reboots for a resize, or a disk fails, or the kernel panics, the product is down. Vertical scaling improves throughput and does nothing at all for availability.

Adding boxes

The other answer is more vans. Now the parcels have to be split between them, which needs somebody to decide who gets what, and the drivers need a shared depot rather than a pile in the back of one van.

Horizontal scaling puts a load balancer in front of several identical instances and spreads requests across them. The ceiling moves a long way out — you add capacity in units of one machine, at a linear price — and availability improves as a side effect, because losing one instance out of four costs you a quarter of your capacity instead of all of it.

The bill comes due as coordination. Any request can now land on any instance, so an instance must not hold anything the next request needs. A session in memory breaks the moment a customer’s second request lands elsewhere, so sessions move to a shared store; an in-process cache stops being coherent, because four copies drift apart; a lock stops working, because it only excludes the threads inside one process. This is what “stateless” means in practice and why REST insists on it: keep the request’s state in the request and the durable state in a store everything can reach. Operationally you now have deploys that roll across a fleet, logs from four places, and a real distributed system with real partial failures.

The scene runs the same six requests twice — once onto a single box, then across three.

The big box’s badge counts requests that reached it, not requests it kept up with; the last two arrive at a machine that is already at its limit. In the second era the same six requests are split two apiece, and two of those requests reach down to the shared session store — the hop the single box never made, and the price of any instance being able to serve anybody.

Which one, and when

Vertical first, almost always. It is one command, it buys real time, and time is what you actually need while the product is still changing shape. The mistake is not choosing vertical; it is choosing vertical and letting the code grow assumptions that only hold on one machine.

So the useful discipline is to scale up while building as though you will scale out: no session state in process memory, no local files that matter, no background job that must run in exactly one place by accident rather than by design. Those are cheap while you have one instance and expensive to retrofit across four. Twelve-factor apps are largely a list of this habit.

The arithmetic that decides the moment fits in a few lines:

/** Requests a second one instance of this app handles. Measure it — the
 *  number is yours, and nobody else's is worth borrowing. */
const PER_INSTANCE_RPS = 400;

/** The same measurement on the largest instance your provider rents. */
const LARGEST_INSTANCE_RPS = 800;

/** Negative means no single machine is big enough: the vertical road ends. */
function verticalHeadroom(peakRps: number): number {
  return LARGEST_INSTANCE_RPS - peakRps;
}

/** How many equal instances that same peak needs behind a load balancer. */
function instancesNeeded(peakRps: number): number {
  return Math.ceil(peakRps / PER_INSTANCE_RPS);
}

const peakRps = 1_200;
const headroom = verticalHeadroom(peakRps); // -400: nothing left to buy
const fleet = instancesNeeded(peakRps); //      3: three boxes at 400 rps each

Both numbers come from one measurement, which is the part worth taking seriously. PER_INSTANCE_RPS is not a figure to copy from an article; it is what your application does on your hardware with your traffic mix, and every decision above is downstream of measuring it honestly.

One more number is hiding in that sample. Three boxes at 400 rps is exactly 1,200, so the fleet covers the peak with nothing to spare. Lose one box at half past nine and 800 rps of capacity meets 1,200 rps of traffic: a third of the load has nowhere to go, at the busiest hour of the day. Horizontal scaling only buys availability when the fleet is sized for a failure.

/** How many equal instances the peak needs, with `canLose` of them down. */
function instancesNeeded(peakRps: number, canLose = 0): number {
  return Math.ceil(peakRps / PER_INSTANCE_RPS) + canLose;
}

const capacityAfterLosing = (instances: number, lost: number): number =>
  (instances - lost) * PER_INSTANCE_RPS;

// 800: short of the peak
console.log(capacityAfterLosing(instancesNeeded(1_200), 1));
// 1200: exactly the peak
console.log(capacityAfterLosing(instancesNeeded(1_200, 1), 1));