Answers · Putting a model into service
What's the difference between searching and just filtering a database?
In shortA filter returns every row that matches a condition you stated; a search returns a short ranked list of things that might answer you, so ordering, not matching, is the job.
A filter returns every row that satisfies a condition you stated, in no particular order. A search returns a short ranked list of items that might be relevant, and the order is where the information lives. The first answers a question you already knew how to ask precisely. The second is a guess about which few of many things deserve attention, and it is allowed to be wrong about everything except which piece mattered.
Take a question typed into a contract search box: how do I get out of this deal early. The clause that answers it reads termination for convenience. The two share no content words at all: get, out, deal, early on one side, termination, convenience on the other, an empty intersection. A condition requiring the query's words to appear matches 0 of the 4,000 clauses in the collection. A ranked search over the same 4,000 scores all of them and keeps 20, with that clause at rank 6. Six of twenty is the top 30 per cent of the list, which is a usable position. It matched on nothing and was found anyway, because the comparison was between meanings rather than strings.
Why order, and not membership, is the output
A filter returns a set, which is to say one equivalence class: everything in it is equally good and the sequence is incidental. A search returns a sequence, and the sequence is the answer. If the right clause is returned at position 14 of 20 and the reader only uses the first five, the reader has effectively not received it. In one tally of a hundred wrong answers, 58 came from text that never reached the shortlist, 27 from text that was there and was buried, and 15 from text that was read and still misunderstood. Those three sum to 100, and 58 divided by 15 is about 3.9, so nearly four times as many failures come from material that was never nominated as from material the model misread. Blaming the model is aimed at the smallest slice.
The shortlist sets a ceiling rather than making a contribution. With twenty pieces returned, the answering piece is somewhere in the list for 84 questions in 100, and 72 in 100 are answered correctly. The reader therefore converted 72 of the 84 chances it was given:
The instinct is to treat the box as a WHERE clause, and when it comes back empty, to widen it: add ORs, drop a word, try a synonym by hand. That is the right repair for a filter, whose only failure is being too narrow, and it is the wrong repair here for two reasons. First, a document can answer a question while containing none of its words, so no amount of widening a lexical condition reaches it; widening only admits more documents that share vocabulary and are still not the answer. A condition can put a thing in or leave it out, and relevance is a matter of degree, so the honest output is an ordering. Second, the failure is quiet. An over-narrow filter returns zero rows and announces its failure. A search that puts the right clause at rank 40 of 40 returns a full page that looks perfectly healthy.
Where the condition wins
Filtering is not the inferior operation. It is the correct one when the question is already exact: invoices over 10,000 pounds dated March, rows where status is failed, records with no owner. There the condition is the whole request, completeness matters, and any ordering would be arbitrary. The test is whether a machine could check a candidate answer, rather than merely score it. If it could, a condition gives you certainty for the price of a scan, while a ranking gives you a probability and a bill for comparing every document in advance. The two compose well in one direction: filter on metadata first to shrink the collection, then rank what survives.
Shortlist length is the knob most often confused with quality. Going from 20 pieces to 100 lifts the share of questions whose answer is somewhere in the list from 84 in 100 to 93 in 100, nine points, and lifts correct answers from 72 to 78, six points, while multiplying the reading cost by five. The nine points are real recall. The six are what survives being buried among ninety-nine neighbours. On the curve that models both effects, ten pieces give presence and use, so ; twenty pieces give presence and use, so . The second number is the one you are paid for, and it is already falling while the first is still climbing.
Worth keeping
- A filter returns all rows that satisfy a condition; a search returns a ranked few, and the ranking is the information.
- A document that was never returned cannot be recovered by any later stage, while an irrelevant one that was returned can be dropped.
- Before blaming the model for a wrong answer, check whether the text that would have answered the question was in the shortlist at all.
Where this goes next
This question is one step of Finding Things by Meaning, a course that works through its subject from the beginning. The page that teaches this part of it is The Question a Retrieval System Answers, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why do I still need exact keyword search when I'm searching by meaning?Because meaning matching compresses text into a few hundred numbers and need not keep a rare part number or error code intact; exact keyword matching weights rare strings so heavily it finds them immediately.
- Why does the model return JSON wrapped in extra text when I asked for JSON?A request for JSON does not force the model to return only JSON, so you need to validate the reply and repair it once rather than assume it will parse.
- Why do the numbers look good while users keep complaining?Because the score measures a fixed set of cases with a fixed rule, and the users are living through behaviour that the set and the rule never covered.
- How do I make the model answer from the documents I give it instead of its own memory?Fence the documents as labelled reference material, put them before the question, tell the model to prefer them over memory, and require a passage number for each claim.
- Why does the memory used by a model keep growing as the conversation gets longer?The model's weights do not grow. It keeps a cache of keys and values for every earlier token, so memory rises with conversation length and with how many conversations run at once.
- How many legitimate requests should I be willing to block in order to catch the bad ones?Block no more legitimate requests than one prevented harmful answer is worth: the break-even count is the harm cost divided by the refusal cost.