Run Both on the Same Cases and Most of the Noise Disappears
Last timeIs That a Real Improvement
The single technique that buys the most in evaluation: compare per case rather than comparing two averages, and the hard cases stop counting against you.
Why pairing works
The previous lesson left a discouraging number: four hundred cases, six points
before anything can be claimed. Most real improvements are two or three points.
The way out is not more cases. It is to notice where the noise comes from.
When you score version A on one sample and version B on another, the difference
between the two numbers contains two things: the real difference between the
versions, and the difference between the two samples. The second part is
usually much larger. Some cases are simply harder than others, and whichever
version happened to draw the harder sample looks worse.
Run both versions on the same cases and that part is shared. Compare within
each case, and it cancels.
- the uncertainty on the difference, which is the thing you actually want small
- the uncertainty on the first version score
- the uncertainty on the second version score
- how strongly the two versions agree case by case: near 1 when they are similar systems, near 0 when they are unrelated
The part worth sitting with is that the benefit grows as the two versions
become more similar. The situation where independent sampling is most hopeless,
a small prompt change that affects a tenth of answers, is precisely where
pairing helps most.
Reading a win rate
In practice the comparison is not usually two scores subtracted. It is a
verdict per case: A better, B better, or no difference. The result is a win
rate.
The lesson stops here
4 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents