The Length at Which Everything Changes
Last timeThe Cost of Remembering
Past a certain conversation length the model stops being the expensive thing to read, the conversation does, and the arithmetic of looking back stops being free.
The last lesson established that what a model keeps about a conversation grows
in proportion to its length. This lesson is about what that growth does, and the
answer is two different things that are easy to confuse.
Two costs, both growing
The first is capacity. A longer conversation occupies more memory, so fewer
conversations fit, so fewer requests can be grouped together. That was the end of
the last lesson.
The second is speed, and it is separate. To produce one new word, the part of the
model that looks back must read every kept summary, because the new position has
to be compared against all of them. So the movement needed for a single step is
not fixed: it grows with how far the conversation has already gone. A machine
producing the ten thousandth word of a conversation is moving far more bytes per
word than it was at the hundredth.
The two costs are worth keeping apart because they fail differently. Running out
of capacity shows up as requests being refused or queued while the machine looks
underworked. Running out of speed shows up as every reply arriving slowly, with
nothing refused at all. A team watching only one of them will keep being
surprised by the other.
- the weights, read once no matter how many requests are in the group or how long they are
- every conversation's store: the number grouped, times the bytes per position, times how far each has run
That difference between the two terms is the heart of the lesson. Grouping
requests spreads the first term and piles up the second.
The lesson stops here
5 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents