What the Model Actually Sees

Almost nobody has printed the complete input their own feature sends. Doing it once, and accounting for every part, is the start of treating it as a designed object.

Print the whole thing once

Here is an exercise worth doing before reading any further, on whatever feature

you currently have running. Find the place where the input is assembled, add a

line that writes the finished string to a file, run one real request, and read

the file.

Almost nobody has done this. The input exists only as the result of running the

assembling code, and reading that code is not the same as reading its output,

because the code is a set of branches and the output is one specific document.

What the exercise finds, reliably, is some combination of the following: an

instruction contradicting another instruction three hundred lines below it, a

piece of a previous feature nobody removed, a placeholder that was never filled

so the input literally contains the words insert context here, a date that is

six months stale, an example of the desired output that no longer matches the

desired output, and several thousand tokens of a document that was supposed to

be summarised and was pasted in whole.

FIG 1The line that makes the input visible

python

body = assemble(system_text, examples, passages, question)

open("/tmp/last_input.txt", "w").write(body)   # the whole thing, as sent
print(len(body), "characters", count_tokens(body), "tokens")

reply = model(body)

Three lines, and they have to go at the point where the input is finished rather than anywhere earlier, because everything interesting happens during the assembling. A version that logs the pieces separately is much less useful, since the thing worth reading is the document as the model receives it, including the joins between the pieces and whatever the joins did to the formatting.

The pieces and who wrote each

Having printed it, the next step is to account for every part. A typical input

in a working feature has five or six, and they were written at different times

for different reasons.

FIG 2One feature input, broken into its parts

	tokens	share of the input	sent every time	last changed, months ago
the instruction about to	310	9	1	14
rules added after compla	540	16	1	2
three worked examples	620	18	1	9
six retrieved passages	1300	38	0	0
the conversation so far	580	17	0	0
the question	60	2	0	0

The share column is percentages and the last column is how long ago that part was last touched. The second row is the one to look at: rules added one at a time after individual complaints are now a sixth of every request, and nobody has read them as a set. The fourth row is the largest single item and is the subject of the fifth and sixth lessons. The question the person actually asked is two per cent of the document.

The table has an uncomfortable implication. The piece that is most carefully

considered, which is usually the question or the retrieved material, is a small

part of what the model reads, and the majority of the input is historical.

FIG 3Where each part came from

step	the part	who wrote it	when	has anybody read it since
1	the role and tone instruction	whoever built the prototype	at the start	no
2	do not speculate about pricing	a product manager, after one incident	month nine	no
3	always reply in British English	a different person, after a different incident	month eleven	no
4	the worked examples	an engineer, copied from a notebook	month three	no
5	the passage template	the search work, generated	month seven	the code, not the output

5 steps

The last column is the point of the figure. Each of these was a reasonable thing to add at the moment it was added, and the document they now form has no author. This is how an input comes to contain two rules that contradict each other: both were right in isolation and nobody ever saw them in the same reading.

The part you send every time

Of the pieces, some vary with the request and some do not. The fixed part has a

property the rest lack: it is multiplied by every single request you serve.

FIG 4What a daily budget affords for the fixed part

: what the fixed part may cost per day
: requests per day
: the price of one input token
: the length of the fixed part, in tokens

The expression is worth running in both directions. Forwards, it says what an existing preamble costs: fourteen hundred tokens across four thousand requests is five and a half million tokens a day, for text nobody has read in a year. Backwards, it puts a length on what the fixed part is allowed to be, which turns an open ended document into something with an edge. Note that caching changes the price of the repeated part substantially where it is available, which makes the fixed part cheaper rather than free.

This is the argument for the instruction lessons that follow being about

brevity as well as clarity. A rule that earns its place is worth sending four

thousand times a day. A rule added after one incident, covering a case that has

not recurred since, is being sent four thousand times a day as well.

Tokens rather than words

The last thing to get right is the unit. Models read, and bills count, pieces

of words rather than words, produced by a scheme that splits common sequences

into single pieces and rare ones into several.

For ordinary English prose the ratio is roughly four characters to a piece, so

a thousand words is about thirteen hundred pieces. The ratio is not stable. Code

is worse, because punctuation and indentation each count. Identifiers in code

are worse again, since an unusual variable name may be five pieces. Languages

other than English are usually worse, sometimes by a factor of two or three for

the same meaning. Long numbers, identifiers and dates are all expensive.

FIG 5The same meaning, measured four ways

	words	characters	tokens	characters per token
a paragraph of English p	120.0	680.0	160.0	4.3
the same text in a langu	96.0	720.0	310.0	2.3
a function in Python	45.0	520.0	190.0	2.7
a list of forty order id	40.0	480.0	320.0	1.5

The last row is the one that catches people. A list of identifiers is short to look at and expensive to send, because each identifier is split into several pieces, and a feature that pastes in a few hundred of them is spending far more than the character count suggests. The practical rule is to measure rather than estimate, which is the second thing the printing exercise at the start of this lesson should produce.

The next lesson is about the largest fixed piece in most inputs, which is the

instructions, and about why a rule that reads well often fails in practice.

Recap

The input to a model in a real feature is assembled by a program from several pieces, most of which were written at different times by different people for different reasons.
Printing the whole assembled input, exactly as it is sent, is a half hour of work that reliably finds something nobody meant to be there.
The fixed part is sent on every single request, so its length is multiplied by your traffic and is the one piece worth measuring before anything else.

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

NextInstructions That Survive Contact →