Answers · Putting a model into service

Why do I still need exact keyword search when I'm searching by meaning?

In shortBecause meaning matching compresses text into a few hundred numbers and need not keep a rare part number or error code intact; exact keyword matching weights rare strings so heavily it finds them immediately.

You still need exact keyword search because a meaning search is built to put a piece near things on the same subject, not to preserve an eight character identifier. A rare part number or error code is exactly the kind of token that a keyword index weights most heavily, so it can find in one step what a meaning search may never surface.

Take a query for part MX-4471B. The manual has a page about MX-4471B. Meaning matching reduces that page to a few hundred numbers chosen so the page sits near other pages about the same product family, noises, and repairs. Nothing in that reduction was asked to keep one eight character string, and in a collection with four hundred part numbers of the same shape it generally does not. The page may come back as a loose paraphrase about the product family, or not come back at all.

Word matching has a term for exactly this. It scores each piece against each question word, and the score has three parts:

w = \log\dfrac{N}{n} \times \dfrac{f\,(k+1)}{f + k\,b}

The first factor is rarity. If a word occurs in one piece out of a million, its rarity term is $\log(1{,}000{,}000/1)=\log 1{,}000{,}000\approx 13.8$ ; if it occurs in half the collection, the term is $\log(1{,}000{,}000/500{,}000)=\log 2\approx 0.693$ . The ratio is about 20, so under the same frequency and length, the rare word contributes twenty times as much. That is why a part number dominates the ranking when it appears. The second factor is how often the word appears, flattened so one extra mention does not run away with the score, with $b$ correcting for pieces of different lengths. None of this is learned. It is counted from the collection you already have, so it has no notion of meaning but also no domain mismatch.

Why the natural guess fails

The natural guess is that a good meaning search subsumes keyword search: if it captures meaning, exact matching is just a special case. It is not. Compression to a few hundred numbers is lossy, and the loss is not aimed at identifiers. Two strings that differ in one character can sit close together in the vector because they occur in similar contexts, while the exact one you asked for has no privileged position. A representation built from a few hundred numbers has no obligation to preserve one unusual token, so a rare part number is not a special case the model handles better; it is the case the model is least equipped to handle.

Why merging on rank works

The two methods fail on different questions, so you run both and merge their ranked lists. The merge uses positions, never raw scores, because the two score scales mean nothing to each other. A simple merge score is one divided by a constant plus the rank, summed across the lists a piece appears in:

\text{score}(d) = \sum_{L} \frac{1}{60 + \operatorname{rank}_L(d)}

With that constant, a piece at ranks 1 and 34 gets $1/61+1/94=0.0164+0.0106=0.0270$ . A piece at ranks 8 and 3 gets $1/68+1/63=0.0147+0.0159=0.0306$ , so it beats the first despite never being rank one: agreement between two independent methods is evidence. Ranks 2 and 41 give $1/62+1/101=0.0161+0.0099=0.0260$ ; ranks 50 and 1 give $1/110+1/61=0.0091+0.0164=0.0255$ . A piece at rank one in one list and absent from the other still gets $1/61=0.0164$ , enough to reach a shortlist of fifty. The merge has not picked a final answer; it has put all four candidates into a shortlist where a later stage can look properly.

Where it stops helping

In a hundred questions a meaning-only system failed, 46 would have been found by word matching alone, 23 failed because the piece was cut badly, 19 needed reasoning across pieces and 12 had no answer in the collection at all. The counts sum to 100: 46+23+19+12=100. Adding keyword search recovers the 46, which is nearly half, but it cannot recover the 23 cutting failures or the 19 cross-piece questions, and nothing recovers the 12 absent answers. If every useful query is a paraphrase and the collection has no rare strings, the rarity term has little to bite on and the balance shifts towards meaning matching. Exact search still earns its place whenever identifiers, names, codes or quoted phrases must survive.

So the two searches are not rivals with one due to replace the other. Keep the exact term path alive because it is the only part of the system that gives a rare string enormous weight, run the meaning path because it is the only part that handles paraphrase, and merge on rank because the failures fall on different questions. The second search costs one extra query, and the merge is the part that recovers failures neither method reaches alone.

Worth keeping

A representation built from a few hundred numbers has no obligation to preserve one unusual token, so a rare part number can be lost by meaning matching.
Word matching scores a piece by how often the question's words appear, weighted by how rare each word is in the collection; a string appearing in one piece out of a million gets an enormous weight.
Merging two ranked lists using ranks rather than raw scores recovers most of both methods because their failures fall on different questions.

Where this goes next

This question is one step of Finding Things by Meaning, a course that works through its subject from the beginning. The page that teaches this part of it is Exact Words and Rough Meaning, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course