Why Waiting Explodes Near the End

Last timeHow Busy Is Busy

Waiting time does not rise in step with load. It rises gently and then vertically, so the last ten per cent of capacity costs more than the first ninety.

The shape of the curve

The single most useful fact in this subject is that waiting time does not rise

in step with load.

The reason is visible in one expression. The waiting is proportional to the

load and inversely proportional to the spare capacity, which is one minus the

utilisation. As the utilisation climbs towards one, that spare capacity shrinks

towards nothing, and dividing by something approaching nothing produces a

quantity approaching everything.

FIG 1How long work waits before it is served

: the utilisation, a share between nought and one
: the service time
: the typical time spent waiting before service starts

Written this way the behaviour is obvious: the numerator rises gently from nought to one as the system fills, and the denominator falls from one to nought, so the ratio runs away. At half load the wait is one service time. At nine tenths it is nine. At ninety nine hundredths it is ninety nine. The expression is exact for a particular pattern of arrivals and approximate for everything else, and the shape it describes holds whatever the pattern is, which is why it is worth carrying around even though the constant is not.

FIG 2Waiting time against utilisation, for a service taking forty milliseconds

time spent waiting

This is the figure to carry in your head, and the thing to notice is where it bends. From nought to one half the curve is nearly flat and the wait is at most one service time, so the first half of the capacity is almost free. From eight tenths onward it climbs steeply, and past nine tenths it is effectively vertical, so a ten per cent increase in traffic at that point multiplies the waiting rather than adding to it. The same increase at four tenths would be invisible, which is why capacity arguments conducted without this curve go wrong.

Waiting at a given load

The wait is a multiple of the service time, which makes it portable between

systems of completely different speeds. A service taking fifty milliseconds at

nine tenths load waits nine times fifty, which is four hundred and fifty

milliseconds, and a batch job taking five minutes at the same load waits forty

five minutes.

That multiple is the useful currency. It means a fast service at high load and

a slow service at low load can produce the same waiting, and it means an

improvement in service time helps twice over: once because the work itself is

quicker, and again because the lower utilisation that results shortens the

queue.

FIG 3The waiting multiple at each level of load

	waiting, in service time	wait for a forty millise
half loaded	1	40
two thirds loaded	2	80
four fifths loaded	4	160
nine tenths loaded	9	360
ninety five per cent	19	760
ninety nine per cent	99	3960

The first column is the whole lesson in six numbers and is worth memorising, because it converts a utilisation figure into an expectation about waiting without any arithmetic at the moment it is needed. The step from nine tenths to ninety five per cent is the one that surprises people: five percentage points of load roughly doubles the waiting, because what actually halved was the spare capacity. The last row is what an incident looks like from inside.

Working backwards from a target

The relation is more useful run backwards. Most teams do not choose a

utilisation because they have a view about utilisation; they have a promise

about response time, and the utilisation that promise allows is a consequence.

Rearranged, the utilisation is the acceptable wait divided by the sum of that

wait and the service time. A service taking forty milliseconds that may wait no

more than sixty milliseconds is limited to a utilisation of sixty over a

hundred, which is nought point six. That is a much more defensible number than

a target chosen by convention, and it has an argument attached to it that

survives contact with somebody asking why the machines are not busier.

The lesson stops here

1 more paragraph to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents