The Stage That Can Afford to Be Slow

Last timeBuying Speed With Accuracy

The first stage compared two representations made in ignorance of each other. A second pass that reads the question and the piece together is far better, and only affordable on a short list.

The first stage had an awkward constraint imposed on it, and it is worth naming.

Each piece of the collection was represented once, in advance, before anybody

knew what would be asked. So that representation has to serve every possible

question, which means it is a compromise across all of them. A page covering

delivery charges, returns and warranty terms cannot emphasise the delivery part

for a question about delivery, because at the time it was represented there was

no question.

A cross reader has no such constraint. It is given the question and one piece at

the same time, so it can weigh every word of one against every word of the other.

It can notice that the question is about returns and that the paragraph on

returns, three lines deep in a page about several things, answers it exactly.

That comparison is strictly richer than comparing two lists of numbers made in

ignorance of each other, and the price is that nothing can be precomputed.

FIG 1What happens after the first stage hands over its shortlist

Everything in this diagram is proportional to the shortlist length and independent of the collection size. That is the property that makes the stage affordable, and it is also exactly why it cannot be the first stage.

FIG 2Time for one question, with both stages

: pieces in the collection, or in practice the candidates the index examined
: cost of one representation comparison, a few hundred multiplications
: candidates passed to the second stage
: cost of one cross reading, which is thousands of times c_{1} and cannot be precomputed

With a hundred and twenty candidates examined and fifty cross readings, the second term dominates the first several times over. Cross reading the whole collection would multiply that second term by two hundred thousand, which is why nobody does it and why the arrangement has two stages rather than one.

FIG 3One shortlist, before and after the second pass

step	position from the first stage	position after reranking	score from the cross reader	what happened
1	14	1	0.94	The page that actually answers the question. Fourteenth out of fifty, which no reader would have used.
2	1	6	0.41	The piece the first stage liked best. It shares vocabulary with the question and does not answer it.
3	3	2	0.77	Genuinely useful background. Both stages agree it belongs, which is the common case.
4	38	3	0.71	Near the bottom of the first list and third after reranking. Reranking buys little unless the first stage was told to hand over enough candidates for cases like this.

4 steps

The last row is the practical lesson. A second pass over ten candidates would never have seen this piece. The first stage should be generous precisely because something downstream is able to sort out the mess, and it can only sort out what it was given.

FIG 4The same fifty candidates, reordered four ways

	milliseconds added per q	right piece placed first	question answered correc
no second pass	0.1	41.0	67.0
a small dedicated cross	38.0	63.0	79.0
a larger dedicated cross	52.0	68.0	82.0
a general language model	840.0	71.0	84.0

The marked row shows the familiar shape: the most expensive option is the best and the margin is small. A purpose built cross reader at thirty eight milliseconds captures most of the available gain, and the general model costs twenty two times as much for two further points.

Placing, not just ordering

Having ordered the candidates, there is one decision left, and it is the one most

systems make by accident.

FIG 5How often the answer is used, by where the right piece sits in the prompt

answered correctly

The same piece, with the same content, answering the same question, is used far more reliably at either end of the list than in the middle. The dip is large enough to matter: a piece at position ten is used about half as reliably as the same piece at position one.

Two consequences follow. The first is to place the best piece first, which is

obvious once the curve is in front of you and is nevertheless frequently not done,

because the code passes the shortlist along in whatever order it arrived.

A refinement used in practice is to put the two strongest pieces at the two ends

rather than the two strongest at the front. The curve says both ends are good

positions, so spending them on the two best candidates wastes neither. Whether

this is worth the oddity of the resulting prompt is a judgement call, and it is

worth measuring rather than assuming on your own system.

The second is to keep the list short. Four pieces have no middle to get lost in.

Twenty do. Since the second pass has just told you which four are worth having,

the information needed to make the list short is already in hand, and discarding

the rest is the point of having ordered them.

The lesson stops here

1 more paragraph to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

The Stage That Can Afford to Be Slow

Placing, not just ordering

1 more paragraph to go

This is the reading half

Read alongside