The Top of the Table Is Not Where the Best System Is
Last timeIs the Gap Real
Take enough noisy measurements and keep the largest one and you have measured luck. Leaderboards do this by design, and a test set reused for a year stops being a test set.
A leaderboard has six hundred entries. The top one scores 92.8. The question
worth asking is not whether that system is good but what the number 92.8 is a
measurement of, and the answer involves the other five hundred and ninety nine
entries.
Why the top of a table is inflated
Every entry's score is its true quality plus a draw from the uncertainty of the
previous lessons. Most entries draw something small. A few draw unusually well.
When you sort the table and look at the top, you are looking at whichever entry
had the most favourable combination of being good and being lucky, and you
cannot tell from the number which part contributed more.
- the measured score of the top entry
- the true quality of that entry
- the standard error of a single measurement
- how many entries were measured
| 100 questions | 400 questions | 1600 questions | |
|---|---|---|---|
| 4 entries | 6.2 | 3.0 | 1.5 |
| 16 entries | 8.7 | 4.2 | 2.1 |
| 64 entries | 10.7 | 5.2 | 2.6 |
| 256 entries | 12.3 | 6.0 | 3.0 |
Notice what this does not say. It does not say the top entry is bad, and it does
not say the ranking is random. Quality still matters and the good systems still
cluster near the top. What it says is that the number attached to the winner is
not a measurement of the winner, and that the difference between first and fifth
place on a crowded table is usually smaller than the noise that sorted them.
The lesson stops here
5 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents