What Splitting a Job Across Model Calls Buys, and What It Charges
Four model calls in place of one doubles the price and more than doubles the wait, so each boundary has to earn that, and only three kinds of boundary do.
A customer writes in about an order. Somewhere a service has to work out what
they are asking, find the order and the policy that governs it, write a reply,
and satisfy itself that the reply does not promise something the policy forbids.
There are two ways to build that, and they are the two this whole subject is
about. One call can be given the question and the customer's recent orders and
asked to produce the reply. Or there can be four calls, one for each of those
four jobs, with the output of each handed to the next.
The second is the default now, and it is usually chosen without anybody pricing
it. So the price comes first, and the argument after it.
What one call costs
Fix the numbers once, because everything that follows uses them. Each step of the
split version sends 1,500 words of instruction and context and gets back 300.
Input is priced at 3 units per million words and output at 15 per million. Each
step takes 0.8 seconds.
| step | what is priced | words | price for a million words | cost in units | what happened |
|---|---|---|---|---|---|
| 1 | one step, words in | 1,500 | 3 | 1,500 times 3 over a million, so 0.0045 | The step's own instructions and whatever context it needs. The 1,500 is not a quarter of the job's instructions, which is the fact the rest of this lesson turns on. |
| 2 | one step, words out | 300 | 15 | 300 times 15 over a million, so 0.0045 | A fifth of the words at five times the price, so the two halves of the bill come out level. That is an accident of these numbers rather than a rule, and a step that reads a great deal and answers briefly tips it the other way. |
| 3 | one step, the whole bill | 1,800 | 3 in, 15 out | 0.009 | Add the halves. This is the unit every argument in this lesson is priced in, alongside the 0.8 seconds a step takes. |
So a step costs 0.009, the four step version costs 0.036 a request, and it
takes four lots of 0.8 seconds, which is 3.2.
The single call is given everything at once and writes the whole reply, so it
sends 3,000 words and returns 600. That is 3,000 times 3 over a million, which
is 0.009, plus 600 times 15 over a million, which is 0.009 again. So 0.018 a
request, and it takes 1.4 seconds, which is longer than any one step and far
less than four of them.
| calls | words in | words out | cost a request | seconds to answer | |
|---|---|---|---|---|---|
| one call | 1.000 | 3000.000 | 600.000 | 0.018 | 1.400 |
| four steps | 4.000 | 6000.000 | 1200.000 | 0.036 | 3.200 |
The reason the input doubled rather than quartering is the thing worth sitting
with. A step's 1,500 words are not a quarter of the job's instructions. They
are that step's own instructions, which the other three do not need, plus
enough of the context to do the work, which the other three mostly also need.
Context is copied across a boundary, not split by it.
| step | step | words sent in | cost so far | seconds so far | what happened |
|---|---|---|---|---|---|
| 1 | sort the request | 1500 | 0.009 | 0.8 | The question and the sorting instructions. None of the order is needed yet, and none of it is sent. |
| 2 | find the order and policy | 1500 | 0.018 | 1.6 | The question again, because this step has to read it, plus the candidate orders and the policy index. |
| 3 | draft the reply | 1500 | 0.027 | 2.4 | The question a third time, the chosen order, the clause, and the house style. Half of this was sent to step one. |
| 4 | check the draft against the policy | 1500 | 0.036 | 3.2 | The draft and the clause. The cheapest step to write and the one that exists purely so something can say no. |
Reason one, the step is not really a model call
Finding the matching order means reading a record. No amount of instruction puts
a customer's order number into a prompt that was not given it, and no generation
produces the delivery date for order 41822 except by having been handed it.
That is the first boundary worth having, and it is not really a choice. The work
leaves the model there. The decision left to you is only whether your own code
holds the control while it happens, calling out to the database and then calling
the model again, or whether the model holds it and asks for the lookup as a tool
call inside one generation. Both pay the same boundary. The first is easier to
log and to stop, which matters when a step has a side effect that cannot be
undone.
def answer(question, customer_id):
kind = classify(question) # step one, a model call
orders = db.recent_orders(customer_id) # not a model call at all
match = pick_order_and_policy(kind, question, orders) # step two, a model call
draft = write_reply(question, match.order, match.policy) # step three
verdict = check(draft, match.policy) # step four
if not verdict.ok:
return escalate(draft, verdict.reason)
return send(draft)Reason two, the instructions fight each other
The drafting step is told to be warm, to offer the nearest thing to what the
customer asked for, and to sound like a person who wants to help. The checking
step is told to be literal, to assume the draft is wrong until the clause says
otherwise, and to refuse on anything it cannot tie to a line of policy.
Put both in one prompt and you have asked a single pass to be generous and
sceptical at the same time, in the same breath, about the same sentences. What
usually comes back is the generous version with a paragraph at the end saying it
has been checked.
The separation buys a second thing that is easier to miss. The checking step is
given the draft and the clause and nothing about what the draft was trying to
achieve. It has no attachment to the reply, because it did not write it, and it
cannot quietly reinterpret the clause to let its own sentence through. A pass
that marks its own work and a pass that marks somebody else's are not the same
test, even when the same model runs both.
Reason three, somewhere a refusal can live
Suppose the policy check has to be able to stop a reply. Not advise against it,
stop it.
Inside one call, the stop is a sentence in the instructions, and whether it holds
is a question about the model's behaviour on that request. Behind a boundary, the
stop is a value your code reads. The two lines at the end of the code above are
the entire difference, and they are what the fourth step is for. The same
property is what lets a step be retried, logged with its input and output as a
pair, or sent to a smaller and cheaper model than its neighbours, because all
three need a place where the work is sitting still and somebody else is holding
it.
So the three reasons, said once: the step's work leaves the model, the step's
instructions contradict its neighbour, or the step must be able to stop what
comes after it. A boundary with one of those behind it is worth 0.009 units and
0.8 seconds. A boundary with none of them is a heading in a prompt that has been
charged for as a call.
The reasons that are not reasons
Four arguments get made constantly, and none of them survives the numbers above.
Each step is simpler, so each step is more accurate. The first half is often
true and the second half does not follow, because the system's rate is not the
average of its steps' rates. It is the product. Four steps that are each good 95
times in 100 give a job that is good 81.5 times in 100, which is what compounding
error does and the whole reason to be careful about step counts.
Shorter prompts are cheaper. Each prompt is shorter and there are four of
them, each carrying its own instructions and most of the same context. That is
6,000 words in against 3,000, so the bill went up by exactly the factor the
argument claimed it would come down by.
It will be faster. Four steps in sequence take 3.2 seconds against 1.4.
Splitting a job can only buy time when the pieces do not need each other and can
run at once, and a chain where each step reads the one before it is the case
where that is impossible. Even then the request waits for the slowest piece
rather than the average one.
It is tidier. Separating four concerns is free when the separation is four
functions, and costs 0.009 units and 0.8 seconds each time the separation is a
call. The diagram looks the same either way, which is precisely the trap: a
drawing cannot show you which of its arrows were paid for.
The test to apply to a split, then, is one question per boundary. Which of the
three reasons is this one? If the answer is none of them, the step belongs in
the prompt beside its neighbour, and the two numbers to beat are 0.018 and 1.4
seconds.
What to hold on to
The four step version of the order service costs twice what one call costs and
takes more than twice as long, because context is copied into each step rather
than divided among them and because the steps wait for each other. Three things
justify paying that at a given boundary, and they are work that is not
generation, instructions that cannot share a prompt, and a refusal that needs
somewhere to live. Simplicity per step is not one of them, and compounding
error is why.
Recap
- Splitting one job into four model calls costs 0.036 units a request against 0.018 for a single call, and takes 3.2 seconds against 1.4, because every step re-sends its own instructions and the steps run one after another.
- A boundary between model calls earns its cost in three cases, which are a step whose work leaves the model, a step whose instructions contradict the step before it, and a step that has to be able to stop the work going any further.
- Four simpler steps are not a more accurate system, because the chance that a request comes out right is the product of the per step chances rather than the average of them.
This is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents