Answers · Putting a model into service
Why does the model pick the wrong tool when two of them sound almost the same?
In shortThe model only sees the written description, so when two descriptions overlap it cannot tell the tools apart and confidently picks the more familiar reading.
Because the model never reads either tool's code. It sees a name, a description string and a list of typed arguments, so when two descriptions use almost the same words the distinction between the tools is not in the text it reads. It resolves the ambiguity to the most common reading and makes the call with confidence.
That is the whole interface. The model cannot see the database, the column types, the rate limit or the fact that a date argument must be the first of a month. Anything true but unstated gets guessed, and the guess is not flagged as a guess. A description that says 'gets the account balance' is not wrong. It is incomplete in ways that force the model to invent the missing parts.
Take 400 calls in which 36 named the wrong tool. With and , , so 91 per cent chose correctly. A single call being right is not the interesting number. On a task that needs five calls, the chance every call is right is , about 62 per cent. That leaves about 38 per cent of five-call tasks with at least one wrong tool choice, which is why a respectable per-call rate can still feel unreliable.
The overlap is the whole signal
The natural guess is that the model reads the function body, or recognises from the name what the tool does. It does neither. A name is a string, and a description is a string, and two strings that both say 'search the documents' do not tell the model which index is meant. The model does not become hesitant when the description is vague. It picks the more common reading, which is usually right, and then repeats that choice every time the same overlap appears.
Put two balance tools side by side. One returns the settled balance in minor units; the other returns the available balance in major units. If both descriptions say 'gets the account balance', the model has no signal for settled versus available and no signal for pence versus pounds. It will choose one reading, say pounds, and use it consistently. Every result is then out by a factor of 100 in one direction, and nothing in the response looks like an error. The same shape appears when one tool treats a limit as a maximum and the other as an exact count.
An unknown identifier that returns an empty list is worse than one that returns an error, because an empty list looks like a true answer rather than a failed call. The model sees no failure, so it reports that the customer has no orders. One sentence in the description, saying an unknown identifier is an error and not an empty result, converts a confident false statement into a visible failure. That sentence costs less than the support ticket it prevents.
Fix the set before the sentences
The largest improvement is usually not a better paragraph. It is fewer tools. Four search tools that differ only in which index they consult ask the model to make a distinction that is genuinely fine, and no wording removes that fineness. One search tool with an index argument makes the same distinction a value the program can validate instead of a choice between near-identical options.
Measure wrong choices per tool, not just overall. If two tools are confused in both directions, merge them. If one tool is chosen when another was meant but never the reverse, the problem is usually in the description of the one being over chosen, because it claims more than it does. If a tool is never chosen when it should be, it is probably named after its mechanism rather than the question it answers.
Better wording stops helping when the overlap is real in the domain. If both tools genuinely do the same job against different backends, a sentence saying when not to use one can reduce the error, but it cannot make the two choices different. At that point the tool set is wrong, not the prose. If the tools must remain separate, make the difference an argument that the program checks, such as an index or a mode value, rather than a distinction the model has to infer from two similar names. A description can prevent a systematic error by stating the thing the model would otherwise guess, but it cannot supply a distinction the tool set does not have.
Worth keeping
- A tool description is the only specification the model gets; it never sees the code behind the tool.
- A vague description does not make the model hesitate; it makes the model guess and then repeat that guess.
- If two tools are confused in both directions, merging them fixes more than rewriting either description.
Where this goes next
This question is one step of Models That Use Tools, a course that works through its subject from the beginning. The page that teaches this part of it is Describing a Tool Well, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Can a web page my agent reads make it do something I never asked for?Yes. A fetched web page can contain a sentence aimed at the model, and nothing in the model's view marks it as different from something you wrote yourself.
- Why does the model have to be asked again after the tool hands back its result?Because a tool call is a request, not an answer. The result must be added to the conversation and the model called again to read it and reply.
- How do I know if a failed task was the model's fault or the tool's?You cannot tell from the reply alone: grade the final state of the data, log what each tool actually returned, and assign every failure to the model, a tool, or your own loop code.
- Why does the model return JSON wrapped in extra text when I asked for JSON?A request for JSON does not force the model to return only JSON, so you need to validate the reply and repair it once rather than assume it will parse.
- Why do the numbers look good while users keep complaining?Because the score measures a fixed set of cases with a fixed rule, and the users are living through behaviour that the set and the rule never covered.
- How do I make the model answer from the documents I give it instead of its own memory?Fence the documents as labelled reference material, put them before the question, tell the model to prefer them over memory, and require a passage number for each claim.