Saying Something and Doing Something

A model never does anything. It produces text, and a program you wrote decides whether that text becomes an action, which is where the whole subject lives.

The model only emits text

Start with the thing most easily forgotten once a system is running. A model

takes a sequence of characters and produces a sequence of characters. That is

the whole of its behaviour. It cannot open a file, send a message, charge a

card or delete a row, any more than a book can.

So when a system is described as a model that books flights, what is actually

happening is that the model produced some characters, which a program read,

and that program booked the flight. The flight was booked by the program. The

model said something; the program did something.

This distinction sounds pedantic and it is the most useful idea in the course,

because everything that follows is about the program.

FIG 1Who does what in a system that looks like one thing

	reads the request	decides what is needed	performs the action	can be held responsible
the person	1	1	0	1
the model	1	1	0	0
the program around the m	1	0	1	1
the tool itself	0	0	1	0

Coded 1 for yes and 0 for no. The third row is the one that matters and it is the row nobody talks about: the program around the model is the only thing in the system that both performs actions and can be designed to refuse them. The model decides what it thinks should happen, which is a suggestion, and the tool performs whatever it is handed, which is obedience. Everything in between is code somebody wrote.

The shape of a request

For the program to act on the model output, the two have to agree on a shape.

The model produces something that says which tool and with what values, and

the program parses it.

FIG 2A request and what the program does with it

python

request = {"tool": "lookup_order", "arguments": {"order_id": "A-4192"}}

name = request["tool"]
if name not in TOOLS:                       # the program chose this list
    return error("no such tool: " + name)
check_arguments(name, request["arguments"]) # the program chose these rules
result = TOOLS[name](**request["arguments"])
return result                               # handed back to the model as text

Five lines, and three of them are the program exercising judgement the model has no part in. The dictionary of available tools is a decision. The argument check is a decision. The choice to return an error rather than guess is a decision. A reader who understands that this file is where the system actually lives, rather than the prompt, is most of the way to debugging any system of this kind.

The format itself does not matter much, and the fact that there is a fixed

format matters a great deal. A fixed shape is what makes it possible to say

that a request is malformed, which is what makes it possible to refuse one.

A system that accepts free text and tries to interpret the intent has no way

to refuse anything, because there is nothing it can point at as wrong.

Where the doing happens

In the figure above, exactly one line performs an action: the call into the

tools table. Everything before it is inspection and everything after it is

reporting.

That line is worth being precise about because it is three things at once. It

is the security boundary, since nothing the model says has any effect until

control reaches it. It is the audit point, since a log written there is a

complete record of everything the system did. And it is the place where the

blast radius of the whole system is decided, because the tools in that table

are the complete list of things this system can possibly do.

FIG 3One request, followed through the system

step	step	who is acting	what exists at this point	can it still be stopped
1	the person asks for a refund	the person	a sentence	yes
2	the model emits a request to call refund	the model	some characters	yes
3	the program parses it	the program	a name and some values	yes
4	the program checks the name and the values	the program	a validated request	yes
5	the program calls the refund function	the program	a refund	no
6	the program returns the result as text	the program	a refund and a message	no

6 steps

Five of the six steps are reversible and the fifth is not. Everything the rest of this course is about happens in rows two to four, because that is the entire window in which anything can be prevented. A system that performs the action first and inspects afterwards has given up the only window it had, which is a surprisingly common design and usually an accidental one.

Who is actually in charge

The experience of using one of these systems is of a model that is driving,

and the architecture is the other way round.

The program chooses which tools exist at all, so the model cannot ask for

anything outside that list. It chooses whether a particular request runs, so

a request can be rejected and handed back. It chooses what the tool result

looks like when it goes back to the model, so it can summarise, truncate or

redact. It chooses when the loop ends. And it chooses what happens when

anything goes wrong.

FIG 4Where the time goes in one exchange

: the time from the person asking to the answer arriving
: the time the model takes for one turn
: the time spent executing the tool

One tool call means two model turns, one to request it and one to read the result, so a four point two second exchange with one point one second model turns leaves two seconds inside the tool. Measuring this split early is worth the few lines it takes, because the instinct when a system feels slow is to reach for a smaller model, and the measurement very often says the model was never the problem.

So the model is a component that proposes, and the program is the system that

decides. This has a practical consequence for how these systems are debugged.

When something goes wrong, the first question is not what the model was

thinking. It is which part of the program let that happen, and in almost every

case there is an answer: a tool that should not have been in the list, a check

that was not written, an error returned in a form the model could not act on,

a loop with no limit.

The next lesson writes the loop itself, which is the smallest piece of code

that turns a model into a system.

Recap

A model emits text and nothing else, so every action in a tool using system is performed by ordinary code that chose to perform it.
The request for an action is just a piece of structured output that the program agrees to recognise, which means the program can also refuse it.
Almost everything that goes wrong in these systems goes wrong in the program around the model rather than in the model, and knowing where the boundary is is the first step to debugging any of it.

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

NextThe Loop →