A List, and a Number on Each Item

Start with something that is going to happen but has not happened yet. A coin

about to be flipped, a word about to be generated, a request about to arrive at a

server. The first thing to write down is not a probability at all. It is a list

of the things that could happen.

That list has to satisfy two conditions, and almost every muddle in the subject

is a failure of one of them. It has to be complete: when the thing happens, what

happened must be on the list. And its items must not overlap: exactly one of them

can be what happened, not two at once. A list of the possible outcomes of a coin

flip is heads and tails. A list consisting of heads, tails, and an outcome

described as a high flip is not a list at all, because two of those can be true

together.

Only once there is a list do the numbers go on it.

The second rule is doing more work than it looks. It says the list is complete,

in the language of the numbers rather than the language of the list: if the

weights add to less than one then something that could happen has been left off,

and if they add to more than one then something is being counted twice. Whenever

a set of weights refuses to add to one, the fault is almost always in the list

rather than in the arithmetic.

Rarely is the question about a single outcome. It is about a group of them: will

the request fail in any way at all, will the generated word be a noun, will the

reward be positive. A group of outcomes taken together is called an event, and

its probability is the sum of the weights of the outcomes in it.

On the four outcome list above, the event that the request did not plainly

succeed is the last three items, with total weight six hundredths. Reading it the

other way, as one minus the weight of the first item, gives the same answer with

less arithmetic, and that shortcut gets more valuable the longer the list is.

This is the part that textbooks hurry past and that matters most in practice.

There are exactly two ways to obtain a set of weights.

The first is to count. Run the thing many times, count how often each outcome

occurred, and divide by the number of runs. This is what a measured failure rate

is, what a word frequency is, and what nearly every number in a machine learning

system ultimately rests on. It requires that the thing has happened many times

already, and it requires a willingness to say that those past occasions and the

future one are the same kind of event, which is a judgement and not a

calculation.

The second is to assume. Nobody has flipped this particular coin ten thousand

times. The weight of a half comes from a claim about the symmetry of the coin

itself. This is faster, applies to things that have never happened, and is only

as good as the claim behind it.

Neither source is more respectable than the other, but they fail differently. A

counted weight fails when the conditions change, so that the past occasions were

not the same kind of event as the future one. An assumed weight fails when the

assumption is simply wrong. The error worth naming is treating an assumption as

though it had been counted, which is what happens whenever a number arrived at by

reasoning is later quoted as though it had been measured.

The most common assumption is that every outcome on the list gets the same

weight. It feels like refusing to assume anything, a kind of neutrality, and it

is nothing of the sort.

The reason is that it depends completely on how the list was cut up. Consider a

part that either works or fails. Two outcomes, so equal weights give a half each.

Now notice that failure comes in two kinds, a clean stop and a silent corruption,

and write the list with three items instead. Equal weights now give a third to

working and two thirds to failing. The situation has not changed at all. Only the

list has, and the supposedly neutral assumption has moved the weight on failure

from a half to two thirds.

So equal weights is a definite claim, and it is a claim about the list as much as

about the world. It is a good claim when the items of the list are genuinely

interchangeable, as the faces of a well made die are. It is a bad claim whenever

the list was written by someone thinking about the problem, because the way a

person chops a situation into cases carries their sense of what matters, and the

uniform distribution then quietly encodes that sense as though it were knowledge.

What survives all of this is small and solid. A list. A weight on each item, not

negative, adding to one. Groups of items, whose weight is a sum. That is the

entire apparatus, and every remaining lesson in this course is built from it

without adding anything new.