Four Times the Data for Half the Error

Last timeLearning From What You Saw

The spread of an average shrinks with the square root of the number of draws, which settles averages down, makes precision expensive, and promises nothing about any single draw.

Measure the latency of a hundred requests and take the average. You have a

number. Now do it again tomorrow with a hundred different requests and you have a

different number. The average is not a fact about the service; it is a quantity

with a distribution of its own, and this lesson is about that distribution.

It turns out to be derivable in two lines from what is already in hand. The

expectation of the average is the expectation of a single draw, which follows

from the addition rule and needs no independence. The spread is where the

interesting part is.

FIG 1The spread of an average

: the variance of a single draw, the spread of the thing being measured
: how many independent draws are averaged
: the variance of the average, smaller than the single draw variance by the whole factor n

The derivation is two steps. Variances of independent quantities add, so the numerator has variance n times sigma squared. Dividing a quantity by n divides its variance by n squared. The two factors of n cancel down to one, leaving the variance of the average smaller by a factor of n. Independence is used in the first step and nowhere else.

Variance is in squared units, so the readable version takes a square root, and

the square root is the whole story of this lesson.

FIG 2The readable form, and what it costs

: the typical distance between your average and the truth, often called the standard error
: one draw's own spread divided by the square root of the count, which is the only part of it you can buy

Because the denominator is a square root, uncertainty falls slowly. Going from a hundred draws to four hundred halves it. Going from a hundred to ten thousand divides it by ten. This is why measurement is cheap at first and ruinous later, and why the last factor of two is always the expensive one.

FIG 3Uncertainty against sample size

spread of the average, single draw spread of 20

The curve falls steeply at the left and then flattens into a long approach to zero. Almost all of the available precision is bought in the first few hundred draws; everything after that is paying progressively more for progressively less. Anyone who has watched a test suite go from flaky to reliable to stubbornly flaky again has met this curve.

Notice which of the two quantities in that fraction is under your control. The

single draw spread is a property of the thing being measured, and it is divided

by one. The count is divided by a square root. So a change that halves the

underlying variation is worth the same as a fourfold increase in data, and it is

usually far cheaper. Pinning the machine a benchmark runs on, warming the caches,

or pairing each measurement against a control all attack the numerator, and

experienced experimenters reach for them long before they reach for more

samples.

The lesson stops here

3 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

Four Times the Data for Half the Error

3 more paragraphs to go

This is the reading half

Read alongside