Chance, Spread and What to Expect
The Second Number Every Average Needs
Last timeThe Number You Would Settle For
An expectation without a spread is a number with no error bar, and the reason spread is measured in squares rather than distances is worth deriving instead of accepting.
Two services both answer in a hundred milliseconds on average. The first answers
in ninety nine or a hundred and one, every time. The second answers instantly
most of the time and takes four seconds once in forty. They have the same
expectation and nothing else in common, and anybody choosing between them on the
basis of the average alone is choosing blind.
So a second number is needed: how far from the expectation do outcomes actually
land. Getting to it takes one false start, which is worth taking deliberately
because the false start explains the shape of the answer.
The obvious measure is always zero
The natural thing to measure is the average distance from the expectation. Take
each outcome, subtract the expectation, and average the results.
It is always zero. Not usually, not approximately: exactly zero, for every
distribution there has ever been. The reason is that the expectation is the
balance point, so the weight sitting above it and the weight sitting below it
balance by construction. The outcomes above contribute positive deviations, the
ones below contribute negative deviations of exactly matching total size, and
they cancel.
| lowest value | highest value | expectation | spread | |
|---|---|---|---|---|
| the steady service | 99 | 101 | 100 | 1 |
| the occasional disaster | 1 | 4000 | 100 | 624 |
Since the cancellation is the problem, the fix is to discard the sign before
averaging. There are two reasonable ways to do that, taking the absolute value or
squaring, and the subject chose squaring. The honest reason is that squares have
algebra and absolute values do not: a square can be expanded, differentiated, and
pushed through the rules for expectation, and this makes almost every later
result possible. There is a second reason that is more than convenience, which is
that squaring makes far outcomes count much more than near ones, and in most
applications a rare outcome ten times further out really is much worse than a
common one slightly off.
The lesson stops here
4 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents