The library

How models work

Be able to explain every step between a string of characters and the vectors a model actually consumes, say where the vocabulary came from, and recognise the failures that tokenisation causes rather than the model

How Text Becomes Numbers

A language model never sees letters. Something upstream cuts the text into pieces, looks each piece up in a list, and hands over a row of numbers. This course covers how that list is built, what it costs, and what it breaks.

8 lessons, written and corrected before you arrived. Reading them here needs no account. The first reads the whole way through; the others open and then stop, because a page nobody owns cannot tell who is reading it. Starting the course gives you your own copy, where every idea has problems standing under it and you can ask about any sentence.

Start reading

  1. 01The Layer Nobody Looks AtA model is a long chain of arithmetic, so text has to arrive as numbers, and the two steps that convert it decide more about behaviour than most people expect.
  2. 02Three Places You Could Cutopening onlyCutting text into characters, into words, or into something in between are three different trades between the length of the list and the length of the sequence.
  3. 03Nobody Wrote This Listopening onlyThe list of pieces is built by a loop that repeatedly glues together the commonest adjacent pair, so what counts as one piece is decided entirely by a corpus.
  4. 04The Number Somebody Had To Chooseopening onlyA longer list of pieces shortens every sequence but widens the table and the output layer, and the gain fades while the cost keeps rising.
  5. 05One Row, Looked Upopening onlyEvery entry in the list owns one row of numbers, the model reads nothing but those rows, and the rows are learned during training rather than assigned.
  6. 06What Ended Up In The Tableopening onlyTraining arranges the rows so that pieces used in similar places sit near each other, which makes some attributes line up as directions and imports the corpus biases wholesale.
  7. 07The Row Is Only The Starting Pointopening onlyA single row cannot hold two senses of a word, so the layers above rebuild each position out of its neighbours until the same piece has different numbers in different sentences.
  8. 08Complaints That Are Really About This Layeropening onlySeveral familiar model failures are not failures of reasoning at all but consequences of how the text was cut, and they can be told apart with a simple test.