Chain, Fork, Loop and Crowd, Priced on One Job
Last timeError That Compounds
Four ways to arrange several model calls behind one request, each drawn and each priced on the same order-answering service, with the two numbers that decide between them.
Four arrangements cover nearly every system that puts more than one model call
behind a single request. The calls can run one after another. They can run
beside each other and have their results brought together. One of them can run
again until something is satisfied. Or the same call can run several times at
once and the answers can be compared. Chain, fork, loop and crowd.
Each has a cost that can be worked out before anything is built, a delay that
follows from the same figures, and a reason it exists. All four are priced here
on the same service, which answers a customer's written question about an order
in four steps: sort the request, find the matching order and policy, draft the
reply, check the draft against the policy.
The chain, and the shape this service already has
A chain is calls in a line, each reading what the one before it produced. It is
the shape a split falls into when nobody chooses a shape, and it is what this
service is.
That last point is easy to get wrong, because the fourth step is a check and a
check sounds like a loop. It is not one. There is no arrow from the check back
to the drafting step, so the check can pass the draft or reject it and nothing
else. A step whose outcomes are pass and reject is an ordinary chain step, and
it multiplies into the end to end accuracy exactly like the other three: at 95
right in 100 per step, four steps give 0.815, which is the four step column of
the table the previous lesson derived. What turns a chain into a loop is an
arrow pointing backwards, not the presence of something called a reviewer.
Cost counts calls, delay counts the longest path
The per call figures are fixed for the whole course and every number below is
built from them. Each step sends 1,500 words of instruction and context and
gets back 300. Each takes 0.8 seconds. Input is billed at 3 units per million
words and output at 15 per million. So the input side of one call is
and the output side is
, and one call costs 0.009. Four of them
cost 0.036 and take 3.2 seconds. For comparison, one call doing the whole job
sends 3,000 words and returns 600, which is , and takes
1.4 seconds.
The lesson stops here
7 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents