Answers · The mathematics underneath

Why does the average of many random samples get closer to the true value as I take more samples?

In shortBecause the average of independent draws is centred on the true value at every sample size, and its typical distance from that value shrinks like one over the square root of the number of draws.

Each sample average is itself a random number, but the whole cloud of possible averages is pinned to the true value: it is centred there at every sample size, and its typical distance from the truth shrinks like $1/\sqrt{n}$ . So the estimate does not creep in from one side. It jitters around the right answer with an amplitude that shrinks to nothing.

\mathbb{E}\big[\hat{Q}_n\big] = Q, \qquad \operatorname{sd}\big(\hat{Q}_n\big) = \frac{\sigma}{\sqrt{n}}

Where the square root comes from

Write the estimate as $\hat{Q}_n = (X_1 + \dots + X_n)/n$ , where each draw has average $Q$ and spread $\sigma$ . Averages add, so the average of the estimate is $(nQ)/n = Q$ . That is exact, not a limit: ten draws are centred on the truth exactly as a million are. Spreads do not add that simply, but variances do provided the draws are independent. The variance of the sum is $n\sigma^2$ ; dividing by $n$ divides a variance by $n^2$ , which leaves $\sigma^2/n$ .

\operatorname{Var}\big(\hat{Q}_n\big) = \frac{1}{n^2}\cdot n\sigma^2 = \frac{\sigma^2}{n} \quad\Longrightarrow\quad \operatorname{sd}\big(\hat{Q}_n\big) = \frac{\sigma}{\sqrt{n}}

Put numbers in. With a single draw of spread one, ten draws give a typical error of $1/\sqrt{10} = 0.316$ ; a hundred give $1/10 = 0.1$ ; a thousand give $0.0316$ ; ten thousand give $0.01$ . To halve the error from $0.1$ you need $1/\sqrt{n} = 0.05$ , which is $n = 400$ . Four times the draws for a factor of two. Every further halving costs another fourfold, which is why the curve falls steeply at first and then flattens.

FIG 1

draws n	typical error 1/sqrt(n)	draws needed to halve it
10	0.316	40
100	0.100	400
1 000	0.0316	4 000
10 000	0.0100	40 000

FIG 2The typical error against the number of draws

spread onespread three

The lower curve is a quantity with spread one, the upper one a quantity with spread three; the upper sits three times as high everywhere, since the spread scales the whole curve without changing its shape. Both flatten: between a thousand and two thousand draws little is gained, and the same flattening is present in every problem.

The natural guess is that a large sample washes out error of every kind, and that is what fails. The narrowing is not what makes the average trustworthy; the centring is. If every draw were shifted by a fixed $0.02$ , the average would still narrow at exactly the same $1/\sqrt{n}$ rate, onto $Q + 0.02$ instead of $Q$ . At a hundred draws that offset hides inside a typical error of $0.1$ . At a hundred thousand draws the typical error is $1/\sqrt{100000} = 0.00316$ , so the offset is about six times the noise and the estimate is confidently, precisely wrong. A big sample cannot tell you which of those two situations you are in.

Independence is doing the work

The step from $n\sigma^2$ to $\sigma/\sqrt{n}$ used independence and nothing else. Let every pair of draws share a covariance $c$ , as they do when a simulation carries state from one run to the next or the generator has hidden structure. The variance of the sum gains a term for each pair, and there are $n(n-1)$ of them.

\operatorname{Var}\big(\hat{Q}_n\big) = \frac{n\sigma^2 + n(n-1)c}{n^2} = \frac{\sigma^2}{n} + \frac{n-1}{n}\,c \;\longrightarrow\; c

With $\sigma = 1$ and $c = 0.01$ , a thousand draws give $1/1000 + (999/1000)(0.01) = 0.001 + 0.00999 = 0.01099$ , a typical error of $\sqrt{0.01099} = 0.105$ rather than the independent $0.0316$ . Take a million draws instead and the first term vanishes while the second stays at $0.01$ , so the error settles at $\sqrt{0.01} = 0.1$ and stays there. The average converges to something. It is not the quantity wanted, and more samples will not fix it.

When there is no spread to speak of

The rate $\sigma/\sqrt{n}$ presumes a finite $\sigma$ , and some quantities do not have one. When the tail is heavy enough, a single enormous draw moves the running average more than everything before it: a draw of $10^6$ among ordinary draws of size one shifts the average by $10^6/n$ when it lands, which is $1000$ at a thousand draws and still $100$ at ten thousand. The running average jumps at the biggest sample sizes instead of settling, and it does so while looking from the inside exactly like an ordinary simulation that has produced a confident figure.

The $1/\sqrt{n}$ figure is a typical error across many repetitions, not a bound on the run in front of you, so a single run can wander anywhere inside it and is not obliged to be closer at a larger sample size than at a smaller one. What is reliable is the width of the cloud, and that is the number worth quoting: with spread one, a thousand draws put you within roughly $0.03$ of the answer, and ten thousand within roughly $0.01$ .

Worth keeping

The average of independent draws is centred on the true value at every sample size, so a small run is noisy rather than biased.
Its typical distance from the truth is the spread of one draw divided by the square root of the number of draws, so halving the error costs four times the draws.
Without independence the average still settles, but on the wrong number, and no number of draws removes the gap.

Where this goes next

This question is one step of Answering With Random Samples, a course that works through its subject from the beginning. The page that teaches this part of it is The Average Is the Answer, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course