Answers · The systems around the model

Why does average latency hide the slow requests users actually feel?

In shortWaiting times are lopsided, so the average sits between the common case and the bad case and describes neither, and it is the bad case that users keep meeting.

Waiting times are lopsided. There is a floor at the service time, a common case just above it, and a long thin end where a small share of requests take many times as long as the rest. The average lands in the gap between the common case and the bad case, so it describes neither of them, and it is the bad case that people remember and complain about.

The arithmetic below is done on one measured service, so the numbers can be checked against each other.

FIG 1

waiting (ms)	share of requests at least this slow, per thousand
18	1000
24	500
95	100
420	10
1900	1
52, the average	260

Four numbers describe this service and none of them is another one in disguise. The median is 24 ms, the average is 52 ms, the ninety ninth percentile is 420 ms, and the slowest one in a thousand waits 1900 ms. The average is $52/24 = 2.2$ times what a typical request waits, and less than a tenth of the way from the common case to the bad case that people actually notice.

The natural guess is that a service whose average is acceptable is acceptable, and that the complaints come from a handful of outliers that can be discounted. Both halves of that fail. The mean is not a summary of the experience, it is the centre of mass of a lopsided distribution, so a team can shave the common case, watch the average improve, and leave the end that generates every complaint exactly where it was. And the outliers are not a handful: one request in a hundred on a service handling a thousand requests a second is ten slow requests a second, every second, forever.

What somebody loading a page actually meets

A page that calls twenty backend services and renders only when all twenty have answered. If each call is slow once in a hundred, the page is slow whenever any one of them is, which is easiest to count backwards: the page is fast only if all twenty are fast.

1 - 0.99^{20} \;=\; 1 - 0.818 \;=\; 0.182

Roughly one page load in five contains at least one slow call, even though ninety nine calls in a hundred are perfectly fine. The user does not average their twenty calls and they do not experience the average of the service. They wait for the slowest of the twenty, every time they load the page, and the tail of the service is the ordinary experience of using it.

A request of many parts inherits every tail

Make the parts better behaved and the effect shrinks without disappearing. Suppose each of five calls is slow four times in a thousand, and the whole request needs all five. The chance the request escapes is the chance all five escape, $0.996^{5} = 0.980$ , so it is slow about two times in a hundred, five times the rate of any one part. While the individual shares are small the compounding is nearly additive: $5 \times 0.004 = 0.020$ , which is the approximation the budgeting rule below relies on.

It bends the wrong way as parts are added. Ten ordinary calls each slow once in a hundred give $1 - 0.99^{10} = 1 - 0.904 = 0.096$ , about a tenth of requests slow. Forty give $1 - 0.99^{40} = 1 - 0.669 = 0.331$ , a third, while every one of those forty calls is still meeting its one in a hundred target.

1 - 0.996^{5} \;=\; 1 - 0.980 \;=\; 0.020

FIG 2Chance the whole request is slow, against the number of parts

each part slow four times in a thousandeach part slow once in a hundred

The lower curve is a well behaved part, slow four times in a thousand; the upper one is slow once in a hundred. Both are convex, so each extra dependency costs more than the one before it. A design review that inspects each part separately sees a flat line at 0.004 or 0.01 and approves.

Dividing a promise among the parts

Once the compounding is admitted, a promise about the whole has to be turned into something each part can be built against, and the budget divides rather than being copied. If the whole is allowed to be slow a share $r$ of the time and it depends on $k$ parts, each part gets $r/k$ , because the shares of the parts add up to the share of the whole.

t \;=\; \frac{r}{k} \;=\; \frac{0.01}{20} \;=\; 0.0005

A request allowed to be slow one time in a hundred, built from twenty calls, gives each call a budget of one slow response in two thousand. That is a hard target and the value of the sum is that it says so before the system is built rather than after. Giving every service the same one in a hundred promise as the system is the mistake that fills the gap: twenty parts each meeting one in a hundred compound to about one slow request in five.

The compounding needs the parts to be independent. If one overloaded database makes every call slow at the same moment, the parts do not combine by multiplication, they fail together, which is worse than the arithmetic above and is not fixed by adding replicas of the things that are not the bottleneck.

It also needs the tail to be uncapped. Put a hard timeout on every call and the long thin end is cut off at the limit, the mean and the worst case are then within a factor of one another, and a mean is a defensible summary again. A request with a single dependency has no compounding at all: its tail is just that one service's tail. And a measurement taken on the server side of a queue excludes the queue wait, so it can show a healthy tail while the user is waiting in a line the dashboard never sees.

Averaging percentiles is the measurement error that undoes all of this. A percentile is not a quantity that can be averaged. Take the ninety ninth percentile of each of twenty servers and average them: the slow server's tail is diluted by the nineteen fast ones, and the fleet's real ninety ninth percentile never appears. Roll five one minute percentiles into a five minute figure and the result is not a percentile of anything. Keep the distribution and compute the percentile once, over everything.

Finally, count the requests that never finished. A service under stress sheds its slowest work at the timeout, so the completed requests look fast at exactly the moment the service is failing, and the tail that caused the shedding is the one part of the distribution the report cannot contain.

Worth keeping

A service can improve its average by speeding up the common case while the slow end that users complain about stays unchanged.
A request that depends on many parts is slow whenever any one part is slow, so its slow share is roughly the sum of the parts' slow shares, not the largest of them.
A promise about the slow end has to be divided among the parts, so one slow request in a hundred across twenty calls means one in two thousand for each call.

Where this goes next

This question is one step of Work That Waits in a Queue, a course that works through its subject from the beginning. The page that teaches this part of it is The Average Is Not the Experience, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course