Only the Questions They Disagree About Carry Any Information

Last timeThe Error Bar Nobody Prints

Comparing two systems on the same questions is a far sharper measurement than comparing two separate scores, because everything the questions have in common cancels out.

Two systems, four hundred questions each, scores of 85 and 89. The previous

lesson said those ranges overlap and the comparison is inconclusive. That was

true of the way the comparison was done, and it was the wrong way to do it.

Run them on the same questions

The fix costs nothing and changes everything. Ask both systems the same four

hundred questions, keep the answers side by side, and compare them question by

question rather than score against score.

FIG 1The paired comparison

Nothing here is more expensive than the unpaired version. The same model runs, the same questions, the same marking. The only difference is that the per-question outcomes were kept rather than thrown away after counting, which is a decision about bookkeeping that determines whether the experiment can answer its question.

Where the information lives

Sort the four hundred questions into four boxes.

The lesson stops here

6 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

Read alongside

paperAdding Error Bars to EvalsMiller, 2024Sets out the paired comparison for language model evaluations directly, including why the correlation between two systems on the same questions is what makes the gain possible.arxiv.org

paperWith Little Power Comes Great ResponsibilityCard, Henderson, Khandelwal and others, 2020Works out how large a set has to be to detect improvements of the size routinely claimed, and finds many published comparisons could not have detected what they reported.arxiv.org

paperShow Your Work: Improved Reporting of Experimental ResultsDodge, Gururangan, Card and others, 2019Argues for reporting the distribution of results rather than a single best number, and shows how comparisons change once the effort spent on each system is accounted for.arxiv.org