Answers · Putting a model into service
What does the model actually see when I send a prompt?
In shortIt sees one assembled string: your typed prompt is a small clause near the end of a document that code stitched together from instructions, examples, retrieved passages and earlier turns.
It sees one flat string of text. The sentence you typed sits near the end of it, and everything before that was written by other people at other times and glued together by a program: a role instruction, a set of rules added one at a time after individual complaints, a few worked examples, some retrieved passages, the conversation so far. Nothing in the string marks where one piece ends and the next begins, and nothing tells the model which parts you wrote and which parts somebody wrote in month nine. Sending a prompt is choosing one clause in a document you have probably never read in full.
Print it once
The way to find out is half an hour of work, and it is not reading the assembling code, because the code is a tree of branches and the output is one specific document. Put a logging line at the point where the string is finished rather than anywhere earlier, run one real request, and read the file.
body = assemble(system, rules, examples, passages, history, question)
print(count_tokens(body), "tokens", len(body), "characters")
open("last_input.txt", "w").write(body) # exactly what the model gets
reply = model(body)What turns up is not subtle. Two instructions that contradict each other, three hundred lines apart, each reasonable when it was added. A leftover from a feature that was retired. A placeholder that was never filled, so the input literally contains the words insert context here. An example of the desired output that stopped being the desired output two model versions ago. A document pasted in whole when it was meant to be summarised. None of this shows in a diff, because a diff tells you what changed and the problem is what stayed.
| tokens | share of the input | sent every time | last edited, months ago | |
|---|---|---|---|---|
| the instruction about ro | 310 | 9 | 1 | 14 |
| rules added after compla | 540 | 16 | 1 | 2 |
| three worked examples | 620 | 18 | 1 | 9 |
| six retrieved passages | 1300 | 38 | 0 | 0 |
| the conversation so far | 580 | 17 | 0 | 0 |
| the question | 60 | 2 | 0 | 0 |
Add the first three rows, the ones marked as sent every time, and they come to tokens. That is the fixed part: 43 per cent of this request, and the same 1470 tokens on every request the feature will ever serve.
At one currency unit per million input tokens that is 5.88 per day, which is a month, for text nobody has read in a year. Run the same arithmetic backwards and an open ended document gets an edge: if the fixed part may cost 5 per day at 4000 requests and 1 per million, it may be 1250 tokens long, and every rule you add spends some of that.
The natural guess is that the input is the prompt, plus perhaps a system message you can see in a dashboard. That guess assumes a boundary that does not exist. The pieces are not attached to your prompt, they are concatenated with it, and the model reads the concatenation as one author's text. Nor are the hidden instructions hidden in any interesting sense: the rule about not speculating on pricing reads exactly like the rule about replying in British English reads exactly like your question. The order that ranks them is whatever order the assembling code happens to concatenate in, which is not a decision anybody made about priority. This is also why reading the code instead of the output misleads: the code shows you every rule that could be sent, and you need the one document that was.
Pieces, not words
The unit matters as much as the parts. Models read, and bills count, pieces of words. Ordinary English runs about four characters to the piece: a thousand words of prose is roughly characters, and pieces. The ratio is not stable. A list of forty order identifiers at twelve characters each is 480 characters, but each identifier splits into several pieces, so the list costs about 320 tokens. That is 480 characters over 320 tokens, 1.5 characters per piece, against 4.3 for prose. Short to look at, three times the price per character. Code is worse than prose for the same reason, since punctuation and indentation each count, and text in languages other than English is usually worse again for the same meaning.
Two things move the arithmetic. Caching, where it is available, bills a repeated prefix at a much lower rate, which makes the 1470 fixed tokens cheaper but not free, and it applies only to an exact prefix, so a timestamp at the top of the instruction block destroys it for everyone. And four characters per token is an English prose figure, not a constant, so a character count is not a substitute for a token count on the finished string. The one number worth having before you touch any wording is the token length of the part you send every time.
Worth keeping
- The input a model receives is a single assembled string, and the part a person typed is usually a small fraction of it.
- The fixed part of an input is sent on every request, so its length is multiplied by your traffic and is worth measuring first.
- Token counts are not character counts: a list of identifiers costs about three times as much per character as ordinary prose.
Where this goes next
This question is one step of Writing What the Model Reads, a course that works through its subject from the beginning. The page that teaches this part of it is What the Model Actually Sees, and it reads the whole way through whether or not you have an account.
What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.
Read that pageThe whole courseOther questions people ask about this
- Why does the model return JSON wrapped in extra text when I asked for JSON?A request for JSON does not force the model to return only JSON, so you need to validate the reply and repair it once rather than assume it will parse.
- How do I make the model answer from the documents I give it instead of its own memory?Fence the documents as labelled reference material, put them before the question, tell the model to prefer them over memory, and require a passage number for each claim.
- How can I tell if a prompt change actually helped or if I just got a lucky answer?Compare two versions on the same written set of cases, before and after, and call a change real only when it beats the wobble of that set.
- How do I stop a retrieved document from being read as an instruction?You cannot stop it completely, but fencing each passage in a labelled block, with a line saying it is information to use and not instruction to follow, makes the model far less likely to obey it.
- Why do the numbers look good while users keep complaining?Because the score measures a fixed set of cases with a fixed rule, and the users are living through behaviour that the set and the rule never covered.
- Why does the memory used by a model keep growing as the conversation gets longer?The model's weights do not grow. It keeps a cache of keys and values for every earlier token, so memory rises with conversation length and with how many conversations run at once.