Answers · Putting a model into service

How many cases do I need before the evaluation can even see a small improvement?

In shortAbout four hundred cases per set can see only a six-point move; a two-point improvement needs roughly four thousand two hundred per set.

A set of about four hundred cases can only see a difference of roughly six and a half points between two independently measured scores. If a two-point improvement matters, the arithmetic asks for roughly four thousand two hundred cases in each set.

s=\sqrt{\frac{p(1-p)}{n}},\qquad s_{\text{diff}}=\sqrt{s_1^2+s_2^2},\qquad \Delta_{\min}\approx 2s_{\text{diff}}.

The arithmetic

Take a score near 70 per cent, so $p=0.70$ , and $n=400$ . The standard error on one score is $\sqrt{0.70\times0.30/400}=\sqrt{0.000525}=0.0229$ , which is 2.3 percentage points. Two independent scores each carry that error, and the errors combine rather than cancel: $\sqrt{2.3^2+2.3^2}=\sqrt{2}\times2.3=3.3$ points on the difference. The smallest difference worth claiming is about twice its error, so $2\times3.3=6.5$ points. A three-point move sits inside that, which is why it is not a small result but no result.

n\approx 400\left(\frac{6.5}{\Delta}\right)^2

For a three-point target, $n\approx400(6.5/3)^2=400\times4.69=1{,}878$ , so about 1,900 cases per set. For a two-point target, $n\approx400(6.5/2)^2=400\times10.56=4{,}225$ , so about 4,200 cases per set. A one-point target needs $400(6.5/1)^2=16{,}900$ cases per set. The square is the whole cost: halving the detectable difference takes four times the cases, and going from 6.5 points down to 2 points takes about ten and a half times as many.

FIG 1

cases per set	error on one score	error on the difference	smallest difference worth claiming
100	4.6	6.5	13.0
400	2.3	3.2	6.5
1000	1.4	2.1	4.1
4000	0.7	1.0	2.1

The gut-chosen set

The natural guess is that the evaluation either sees an improvement or does not, and that a disappointing run is bad luck. The set size fixed the resolution before the run. With 400 cases, every difference below 6.5 points is below the resolution, so reporting a three-point move as a small win is the mistake. The other common guess is that the error on a difference is the error on one score. It is not: two independent errors add in quadrature, which makes the difference error larger than either one by a factor of about 1.4. That is why a 400-case comparison cannot honestly separate 71 per cent from 74 per cent.

Where it changes

The calculation assumes two independent sets and a score near the middle of the range. A score near 50 per cent is the worst case, because $p(1-p)$ is largest there: at 400 cases the error on one score is 2.5 points and the smallest claimable difference is about 7.1 points. A score near 95 per cent is measured more precisely, because $p(1-p)$ is much smaller, so 400 cases resolve a smaller difference. If both versions are run on the same cases and compared case by case, the shared variation drops out and the curve is nearer $55/\sqrt{n}$ , so 400 cases can resolve under three points rather than six and a half. The number of cases is therefore not a fixed property of an evaluation. It is a property of the comparison being made.

A dashboard makes the same arithmetic repeat across many numbers. If each score has a one in twenty chance of moving enough to look like a finding, then twenty numbers produce about one apparent finding per look even when nothing changed. Decide the primary number in advance, publish the smallest difference that set can detect next to it, and treat surprises from secondary numbers as candidates for a second run on fresh cases rather than as results.

Worth keeping

At 400 cases and a score near 70 per cent, the standard error on one score is about 2.3 points, the error on a difference between two independent scores is about 3.3 points, and the smallest difference worth claiming is about 6.5 points.
To detect a difference of two points with two independent sets of that kind takes roughly 4,200 cases per set.
Running both versions on the same cases removes shared variation, so the same 400 cases can resolve under three points instead of six and a half.

Where this goes next

This question is one step of Judging a System, Not a Model, a course that works through its subject from the beginning. The page that teaches this part of it is Is That a Real Improvement, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course