Why does latency explode when utilization gets close to 100%?

The natural guess is that wait grows in step with load. If it did, the step from ninety to ninety-five percent would add a small constant. It doubles the wait, from 10 to 20. The step from ninety-five to ninety-eight is three points of load, a tenth of the earlier thirty-point step from fifty to eighty, and it multiplies the wait by 2.5, the same factor the thirty-point step bought. That is the knee: not a change in the machine, but the reciprocal turning a small remaining fraction into a large time.

The reciprocal has a mechanical reading. A backlog of $n$ jobs drains at a net rate of $1-\rho$ jobs per service time, because the server completes one job per service time and new work arrives at rate $\rho$. So the time to clear the backlog is $n/(1-\rho)$. A backlog of 20 jobs at eighty percent load takes $20/0.2=100$ service times to clear; the same backlog at fifty percent takes $20/0.5=40$. The queue length that forms during a burst is what matters, and the drain rate is the headroom.

Little's law ties the queue to the latency: $L=\lambda W$. If a thousand requests arrive per second and each must finish within a tenth of a second, no more than $1000 \times 0.1 = 100$ can be in flight. If the same service lets $W$ drift to half a second, the same arrival rate puts $1000 \times 0.5 = 500$ in flight. The queue depth grows with the latency, so a latency problem is always also a memory and concurrency problem.

Variability makes the reciprocal worse. A burst of twenty jobs arriving at once at eighty percent load has to drain at the spare rate of $0.2$ jobs per service time, taking $20/0.2=100$ service times. The same twenty jobs spread evenly add $0.2$ of a service time each and never form a queue. Two services with identical average load can therefore have queues that differ by the whole burst size, which is why smoothing arrivals buys latency without buying machines.

The explosion is capped when you stop letting the queue hold work. A bounded queue with admission control keeps $W$ finite and turns excess load into rejections, so the failure mode changes from slowness to dropped requests. A deterministic service process halves the waiting time, which becomes $\rho/(2(1-\rho))$; the shape is the same but displaced. At $\rho \ge 1$ the queue never drains and there is no steady state at all: the wait is unbounded and the formula has already told you so through a zero or negative denominator. More servers move the knee to the right but do not remove it, because the reciprocal of the headroom returns.

An operating point near sixty-five percent leaves $0.35$ of capacity free, so the time in system is $1/0.35 \approx 2.86$ service times. At ninety percent it is $10$ service times. The extra idle machines between those two points are not waste; they are the reserve that absorbs a burst and drains the backlog afterwards. Above seventy percent the same reserve is doing both jobs and runs out in both at once.