A Hundred Questions You Wrote Yourself
Last timeReading the Shortlist Properly
Without a set of questions whose answers you know, every opinion about a retrieval system is an anecdote. Building that set is a day of work and it changes what you do next.
Seven lessons of decisions, each of which could have gone another way. Piece
size, overlap, which representation model, whether to merge in word matching, how
to set the index dial, how many candidates to rerank, how many to pass on.
Every one of those is unanswerable without measurement, and almost nobody
measures, because measurement requires a set of questions whose answers you know
and building one feels like a project.
It is an afternoon.
Which numbers
The first is the share of questions whose answer appears anywhere in the
shortlist. It blames the first stage and nothing else, which makes it the most
useful single number in the system.
The second is the share answered correctly. It blames everything, which makes it
the number to report and a poor number to debug with.
A third is worth adding once the first two are stable, because neither of them
can see ordering.
- the position of the right piece in the list returned for question j
- questions in the set
- the average of one divided by that position, which is one when every answer is first and a tenth when every answer is tenth
| answer reaches the short | average of one over the | answered correctly, as a | |
|---|---|---|---|
| meaning matching alone | 84.00 | 0.41 | 66.00 |
| meaning matching with re | 79.00 | 0.63 | 74.00 |
| merged word and meaning | 91.00 | 0.38 | 69.00 |
| merged, with reranking | 88.00 | 0.66 | 81.00 |
How many questions is enough
A measured rate on a hundred questions is not the true rate. It is a sample, and
the gap between the two is often larger than the improvements people announce.
The lesson stops here
4 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents