When a Tool Fails
Last timeChecking What the Model Asked For
Tools time out, refuse and return nothing. The question is which failures are worth another attempt, and what the model should be told about the ones that are not.
Three kinds of failure
A tool call can fail in many ways and they collapse into three groups,
distinguished by one question: could the identical call, made again in a few
seconds, succeed?
A transient failure is one where the answer is yes. A timeout, a rate limit, a
momentary connection error, a service restarting. Nothing about the request was
wrong and the world was briefly unavailable.
A permanent failure is one where the answer is no. The record does not exist.
The customer is not eligible. The argument was rejected. Repeating this call
will produce exactly the same refusal for ever, and the only useful response is
to do something different.
The third group is the one that causes real damage. An ambiguous failure is a
call whose outcome is unknown: the request was sent, the connection dropped,
and nobody knows whether the payment went through.
| could the same call succ | safe to repeat | tell the model | retry in the program | |
|---|---|---|---|---|
| timeout before anything | 1 | 1 | 0 | 1 |
| rate limited | 1 | 1 | 0 | 1 |
| service restarting | 1 | 1 | 0 | 1 |
| record not found | 0 | 1 | 1 | 0 |
| argument rejected | 0 | 1 | 1 | 0 |
| not permitted | 0 | 1 | 1 | 0 |
| timeout after the reques | 1 | 0 | 1 | 0 |
Retrying in the program, not the model
Transient failures are handled by trying again, and where that retry lives
matters more than it sounds.
Inside the program, a retry repeats the byte for byte identical call. Three
attempts with increasing gaps, and if all three fail the failure is reported
once.
Delegated to the model, by passing the timeout back and letting it decide, the
retry is not a retry. The model produces a new request, which is usually
similar and rarely identical: a slightly different search phrase, a rounded
amount, an extra argument it decided might help. For reads this is merely
wasteful. For writes it is how a customer ends up refunded twice, once by the
call that timed out after taking effect and once by the model helpful second
attempt.
- the chance one attempt fails transiently
- attempts needed on average to get one success
Two details make retries safe. Space the attempts out, with a growing gap and a
small random component, because a hundred clients all retrying in unison one
second after an outage are the reason the outage continues. And attach an
identifier to any call that changes something, so that the same identifier
arriving twice is recognised and performed once. That second property is what
turns the dangerous ambiguous failure into an ordinary transient one, and it is
worth having on every tool that writes.
The lesson stops here
3 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents