Build a vocabulary from a corpus rather than downloading one, and predict from it what a model will find cheap, expensive or impossible, well enough to explain a surprising cost, a failed arithmetic answer or a mangled language from the tokenizer alone

Cutting Text Into Pieces

Before a model sees text, something cuts it into pieces, and that choice sets the cost, the context limit and several famous failures. This course builds a vocabulary from scratch and traces what each decision does downstream.

8 lessons, written and corrected before you arrived. Reading them here needs no account. The first reads the whole way through; the others open and then stop, because a page nobody owns cannot tell who is reading it. Starting the course gives you your own copy, where every idea has problems standing under it and you can ask about any sentence.

Start reading

01Two Obvious Vocabularies, Both of Which FailA list of words runs out the moment it meets a word it has not seen. A list of characters never runs out and makes everything long. The answer is between them.
02Count the Pairs, Join the Winner, Repeatopening onlyThe algorithm that builds a vocabulary is four lines long. Start from single bytes, count every adjacent pair, join the commonest into one new symbol, and do it again.
03Working From Bytes Makes the Vocabulary Totalopening onlyThere are too many characters in the world to list, so modern tokenizers start from the 256 bytes instead. That makes nothing unrepresentable, and it makes some scripts very expensive.
04Two Bills That Move in Opposite Directionsopening onlyA bigger vocabulary costs parameters and makes every sequence shorter. A smaller one does the reverse. Adding the two bills gives a curve with a bottom, and that bottom is the answer.
05The Space Belongs to the Word That Follows Itopening onlyAlmost every vocabulary attaches the leading space to the word, so the same word is a different symbol at the start of a line than in the middle. Several famous failures start here.
06Where the Digits Get Cut Decides Whether the Sum Worksopening onlyA long number is chopped into arbitrary groups of digits that do not line up between the two operands. Much of what is blamed on reasoning is this, and it is fixable.
07The Same Sentence, Nine Times the Billopening onlyPrice, speed and context are all counted in pieces, and the same meaning takes several times more pieces in most languages than in English. Here is how much, and why.
08The One Decision That Is Made Before Anything Elseopening onlyThe vocabulary is fixed before the first training step and everything above it is built on the identity of its entries, so changing it later invalidates the model rather than updating it.