How Far Off Is It

Last timeThe Average Is the Answer

An estimate without an error bar is not an answer. The error bar comes from the samples themselves, and it carries an uncomfortable consequence about the cost of precision.

A simulation returns 0.3367. Is that answer good to two digits or to four?

Nothing in the number says. The samples that produced it do say, and getting

the answer out of them costs one extra line.

FIG 1The uncertainty, computed from the same samples

: the standard error: the typical distance between the estimate and the truth
: the spread of a single computed value, measured from the samples themselves
: the number of draws
: the estimate, which is the average of the computed values
: the interval usually quoted, which covers the truth about nineteen times in twenty

Everything on the right comes from the samples already in memory, which is the point: no second simulation, no theory about the problem, no extra draws. The factor of two is a convention and not a law, and it corresponds to the usual nineteen in twenty coverage. Note what the interval claims, which is that intervals built this way succeed about that often. It is not a range the answer moves around inside, because the answer is a fixed number and the interval is the thing that changes from run to run.

FIG 2Typical error against the number of draws

ss = 1

typical error

Turn the knob to change the spread of the quantity being averaged, and watch the whole curve scale without changing shape. Two things are visible and both matter. Moving right along the curve is expensive, since the region past ten thousand draws is almost flat and a great deal of computing buys very little. Turning the knob down is cheap by comparison, since halving the spread does exactly what quadrupling the sample count does. That comparison is the entire argument for the next lesson, which is about turning the knob rather than extending the axis.

FIG 3What each additional digit costs

	draws needed	typical error	seconds at a million dra
one reliable digit	100.000	0.100	0.000
two reliable digits	10000.000	0.010	0.010
three reliable digits	1000000.000	0.001	1.000
four reliable digits	100000000.000	0.000	100.000
five reliable digits	10000000000.000	0.000	10000.000

Each row costs a hundred times the row above it, for a quantity whose single draw spread is one. The first three rows are free, the fourth is a coffee break and the fifth is most of a day on one machine, which is why simulated answers are rarely quoted past three or four digits. The marked cells are where brute force stops being the answer. Reading the table the other way round is the useful habit: decide how many digits the decision actually needs before the run, because two digits is usually plenty and the difference in cost between two and four is a factor of ten thousand.

Sizing a run for a probability

When the thing averaged is one on success and zero otherwise, its spread is

fixed by the probability itself, so the sample size needed can be worked out

before any sampling happens. The hardest case is a probability near one half,

and planning for that is safe for every other case.

Rare events are the opposite problem. What controls the precision is the

number of successes seen, not the number of draws taken, so a probability of

one in a million needs on the order of a hundred million draws before the

estimate means anything. That is the motivating case for the next lesson.

The lesson stops here

1 more paragraph to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

How Far Off Is It

Sizing a run for a probability

1 more paragraph to go

This is the reading half

Read alongside