Describing a Tool Well

Last timeThe Loop

The description is the only thing the model knows about a tool, so a vague one does not produce hesitation. It produces a confident call with the wrong values.

The description is the whole interface

The model cannot read the code. It cannot see the database, the column types,

the rate limit, or the fact that the date argument has to be the first of a

month. It sees a name, a sentence or two, and a list of values with their

types. That is the complete specification as far as the system is concerned.

This is an unusual position to be writing an interface from, and the closest

familiar case is a library used by somebody who will never read its source and

will not ask. Anything true but unstated gets guessed.

FIG 1Two descriptions of the same tool

	says what it is for	says when not to use it	gives units and format	says what failure looks	wrong calls per hundred
get the account balance	1	0	0	0	23
returns the settled bala	1	1	1	1	4

The first row is the description most tools ship with, and it is not wrong, it is merely incomplete in the four specific ways the columns name. The wrong call rates are illustrative and the ratio between them is not unusual. Almost all of the difference comes from the third column: a model that does not know whether the balance is in pounds or pence will pick one, and it will pick the same one every time, so the failure is systematic rather than occasional.

What a good description contains

Four things, and the order is roughly the order of how often each is missing.

What each value means, including its units, its format and its allowed range.

Amounts in major or minor units. Dates as a full date or as a month. Whether a

limit is a maximum or an exact count. Whether an identifier is the internal one

or the one printed on the customer invoice.

When not to use it. A model with two plausible tools will pick one, and a

sentence saying use the other tool when the customer has not yet paid is worth

more than any amount of description of this one.

What it is for, stated in terms of the question it answers rather than the

mechanism it uses. The model is choosing a tool to answer a question, and a

description written as a summary of the implementation does not help with that

choice.

What happens when it fails. If an unknown identifier returns an empty result

rather than an error, the model will treat an empty result as a true answer

and report that the customer has no orders. One sentence prevents this.

FIG 2A description written for a reader who cannot see the code

python

{
  "name": "refund_order",
  "description": (
     "Refund a single order that has already been paid and shipped. "
     "Use check_order_status first if you are not sure of the state: "
     "an order that has not shipped must be cancelled instead, with "
     "cancel_order. Refunds cannot be reversed. "
     "Returns the refund id, or an error naming the reason it was refused."
  ),
  "arguments": {
     "order_id": "the id printed on the invoice, in the form A-0000",
     "amount_minor": "amount in pence, never pounds; must not exceed the order total",
     "reason_code": "one of damaged, late, wrong_item, other"
  }
}

Every sentence here exists because of a specific failure. The pointer to another tool prevents the wrong one being chosen for unshipped orders. The note that refunds cannot be reversed changes how readily the model reaches for it. The units on the amount prevent a hundredfold error that would otherwise be systematic. And the closed list of reason codes converts an invented value into a validation failure rather than a bad row in a table.

Vagueness becomes confidence

The intuition worth correcting is that an unclear description makes a model

hesitant. It does not. It makes the model choose the most common reading of

the ambiguity, which is usually right, and then be wrong in exactly the same

way for ever.

The lesson stops here

2 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents