When the Queue Fills Up

Last timeThe Average Is Not the Experience

When arrivals exceed capacity the only choices are to queue, to shed, or to collapse. Choosing deliberately is the difference between a degraded service and no service.

Shed or queue

The first lesson said that if arrivals exceed capacity the queue grows without

limit. This lesson is about what to do when that happens, because it will.

There are only three outcomes available. Queue the excess, refuse the excess, or

collapse. The third is what happens when the first is chosen without a limit,

and it is worth being precise about why.

Work that waits longer than the client is willing to wait is work that will be

abandoned. The client gives up, and the request is still in the queue, so when a

server eventually reaches it, it does the work and sends the answer to nobody.

That capacity has been spent producing nothing. Worse, the client that gave up

usually retries, so the abandoned request has also generated a new arrival.

Beyond a certain depth, every item admitted makes the service slower without

making anybody happier, and the useful throughput falls while the system is

working as hard as it can.

FIG 1What happens to the useful throughput under overload

share served usefully, with sheddingshare refused, with sheddingshare served usefully, queueing without limit

The first curve is the best anybody can do: the service keeps delivering its full capacity and the surplus is refused, which the second curve measures. The third curve is what an unbounded queue produces, and the shape is the point: it falls away far faster, because the capacity is being spent on work nobody is waiting for any more and the retries from abandoned requests add to the offered load. At three times the offered load, shedding still serves a third of the arrivals and the unbounded queue serves about a tenth, with both machines equally busy.

That collapse is the reason bounded queues are not a detail of tuning. An

unbounded queue does not protect the system from overload, it converts a

recoverable capacity problem into a total failure whose symptoms appear

everywhere except the place that caused it.

How much to shed

Shedding is usually improvised during an incident and it does not need to be,

because the amount is a calculation.

The lesson stops here

5 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

When the Queue Fills Up

Shed or queue

How much to shed

5 more paragraphs to go

This is the reading half

Read alongside