Answers · The mathematics underneath

Why does the average of many random draws get closer to the expected value as I take more?

In shortAveraging divides by the number of draws, so the spread of the average shrinks with the square root of that number; no draw ever compensates for earlier ones.

The draws are not being corrected. The total they are added into keeps growing, and an average divides that total by the count, so the same noise ends up spread over a longer and longer denominator. Stated exactly: an average of $n$ independent draws has spread $\sigma/\sqrt{n}$ , where $\sigma$ is the spread of a single draw. That is the whole reason a running average settles. Not one of the draws knows anything about the ones before it; what changes as you take more is how much any single draw can move the number you report.

\operatorname{Var}\!\left(\frac{X_1 + \cdots + X_n}{n}\right) = \frac{n\sigma^2}{n^2} = \frac{\sigma^2}{n}

Where the shrinking comes from

Two steps do it, and only the first needs independence. Variances of independent quantities add, so the sum $X_1 + \cdots + X_n$ has variance $n\sigma^2$ . Dividing a quantity by $n$ divides its variance by $n^2$ , because variance lives in squared units. One factor of $n$ therefore survives, in the denominator. Put numbers in: with $\sigma = 20$ and $n = 100$ , a single draw has variance $400$ , the sum of a hundred draws has variance $100 \times 400 = 40000$ , dividing by $100^2 = 10000$ brings that down to $4$ , and the square root of $4$ is $2$ . The same answer as $20/\sqrt{100} = 20/10$ . The spread of the average is a fifth of the spread of one draw, and the five is $\sqrt{25}$ -ish arithmetic, not luck.

FIG 1

draws	spread of the average	cost relative to the first row
100	2.0	1x
400	1.0	4x
1600	0.5	16x
6400	0.25	64x

Every halving of the uncertainty costs four times the data. Going from 100 draws to 400 halved the spread; the next halving needs 1,600, and the one after that 6,400, sixty four times the original. Because the denominator carries a square root, the first few hundred draws buy almost all the precision on offer and each further factor of two is dearer than the last. It is usually cheaper to attack $\sigma$ than $n$ : a change that halves the variation of a single draw is worth exactly what a fourfold increase in data is worth, and pinning the machine, warming the caches, or measuring each case against a control does that.

Nothing is owed

The natural guess is that a run of bad luck builds a debt which later draws must repay. It does not, and the reason is that nothing removes the surplus: it stays in the total while the count underneath it grows. Suppose a thousand tosses have produced 550 tails against an expected 500, a surplus of 50 tails. Keep tossing and let everything after that split evenly. After a hundred thousand tosses there are 50,050 tails, still a surplus of 50, but its share of the average is $50/100000 = 0.0005$ rather than $50/1000 = 0.05$ . After a million tosses the surplus is still 50 and the share is $50/1000000 = 0.00005$ .

FIG 2A fixed surplus under a growing count

	tosses	surplus tails	shift in the average
after a thousand tosses	1000.000	50.000	0.050
after ten thousand	10000.000	50.000	0.005
after a hundred thousand	100000.000	50.000	0.001
after a million	1000000.000	50.000	0.000

The middle column never moves, because every later toss is fair and adds nothing to the surplus. The right column is the surplus divided by the number of tosses, and it is the amount by which the average sits above one half. Dilution does all the work.

The deficit is diluted, never repaid. The next toss is a fresh toss with no memory of the thousand before it, and the average converges because a fixed surplus is being divided by a growing count. The same reasoning works in reverse for a lucky start: an early surplus of heads is not a debt on the tails side, it is a term that fades.

A second fact arrives with averaging and is not in the square root formula. As the count grows the distribution of the average approaches a bell shape, almost regardless of the shape of a single draw. Latencies are badly skewed, clicks are ones and zeros, counts are lumpy, and the average of enough of any of them is bell shaped. That is what makes a rule of thumb like two spreads either side of the mean covering about ninety five per cent usable for averages. It too is a limit, and it arrives slowly when the single draw distribution is very lopsided.

Where it stops being true

Two assumptions are doing work. The variance step needs independence: if every draw were the same draw, averaging would change nothing and the spread would stay at $\sigma$ however many you took. And the statement is a limit, not an identity. At $n = 10000$ and $\sigma = 20$ the spread is $20/100 = 0.2$ , small but not zero, and no finite count makes it zero. The expectation also has to exist; a distribution with no finite mean has no value for the average to settle on, and the formula has nothing to divide.

What the guarantee covers is the average over many draws. It makes no claim about any single outcome, gives no prediction for the next one, and does not make a bad run more likely to end. A casino's takings are extremely predictable per thousand bets and entirely unpredictable per bet, and it is this same arithmetic that separates the two.

Worth keeping

The spread of an average of n independent draws is the spread of one draw divided by the square root of n.
Halving the uncertainty of an average costs four times the data, so precision gets expensive quickly.
A surplus from an early run of bad luck is never removed; it is divided by a growing count until it barely moves the average.

Where this goes next

This question is one step of Chance, Spread and What to Expect, a course that works through its subject from the beginning. The page that teaches this part of it is Why Averages Settle Down, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course