What One Step Has to Tell the Next, and What It Must Not
Last timeWho Decides the Next Step
The handover between two model calls is an interface, and carrying the whole history forward instead of a fixed set of named fields nearly doubles what a four step run costs.
Four model calls means four prompts and three handovers. The prompts get
argued over in review. The handovers usually get whatever the first draft
happened to produce, and they are where both the cost and most of the wrong
answers come from.
The system here is the four step one used throughout: a service that answers a customer's written question about an order, in four steps. Sort
the request, find the matching order and policy, draft the reply, check the
draft against the policy. Each step sends about 1,500 words of instruction and
context, gets back about 300, and takes 0.8 seconds. Input costs 3 units per
million words and output costs 15, so one step costs 1,500 times 3 divided by a
million, which is 0.0045, plus 300 times 15 divided by a million, which is
another 0.0045. That is 0.009 a step, so 0.036 and 3.2 seconds for the run.
Hold on to those numbers, because every question below is answered by them.
Free text is a parser nobody wrote
Step one has sorted the request. What does it pass to step two?
The default is prose. Something like "this is a refund request about a late
delivery for order 118254, and the customer sounds annoyed". It reads well, a
person can check it at a glance, and it took no design. It is also a message in
a format with no grammar, sent to a reader that will never admit it could not
read it. Three consequences follow, and all three cost money.
A missing field and an unmentioned field arrive identically. If step one found
no order number, the sentence simply does not contain one. Step two cannot tell
"there was no order number" from "there was one and it was not worth
repeating", and what it does with the ambiguity is resolve it, because
producing the thing that was asked for is the only behaviour it has.
The failure is not reportable. Code that cannot read its input raises. A model
that cannot read its input answers. The parse failure surfaces two steps later
as a confident reply about somebody else's order, by which time all four calls
have been paid for.
And nothing can be checked in between, because there is nothing to check
against. A shape is what a check needs.
from dataclasses import dataclass
@dataclass(frozen=True)
class SortedRequest:
kind: str # one of refund, delivery, change, other
order_id: str # exactly the digits the customer wrote
quoted: str # the customer's own words, not a paraphrase
sure: bool # whether step one settled on the kind
def accept(h: SortedRequest) -> None:
assert h.kind in {'refund', 'delivery', 'change', 'other'}, h.kind
assert h.order_id.isdigit() and len(h.order_id) == 6, h.order_id
assert h.quoted.strip(), 'the customer wording was dropped'That is the real argument for a fixed shape, and it is not tidiness. It is that
a fixed shape can be checked by code, and code is free while a model call is
not. Price the two places a bad handover can be caught. Caught at the first
handover, the run has spent 0.009 and stops. Carried to the end, it has spent
0.036, four times as much, and a reply about the wrong order has been drafted
and possibly sent.
The lesson stops here
7 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents