The library

Putting a model into service

Be able to work out, from a model's size and a machine's published numbers, how fast it can answer, where the time is actually going, and which of the available settings would change the answer

What It Costs to Run a Model

Running a trained model is an arithmetic problem with a small number of terms: how many bits each number takes, how many bytes must be moved for every word produced, how much the kept conversation adds, and how many requests are being served at once.

8 lessons, written and corrected before you arrived. Reading them here needs no account. The first reads the whole way through; the others open and then stop, because a page nobody owns cannot tell who is reading it. Starting the course gives you your own copy, where every idea has problems standing under it and you can ask about any sentence.

Start reading

  1. 01The Two Kinds of WorkA chip that runs a model multiplies numbers and moves numbers, and the second is hundreds of times slower, which quietly decides almost everything else.
  2. 02Paying for Precisionopening onlyEvery weight is stored in some number format, and the choice of format sets both how much memory the model occupies and how fast it can be read.
  3. 03The Ceiling You Cannot Code Aroundopening onlyOne request writing one word at a time must read the whole model from memory for every word, and that alone fixes the fastest it can possibly go.
  4. 04What the Model Holds On Toopening onlyA model consulting the words already seen keeps a summary of each one, and that store grows with every word until it rivals the model itself in size.
  5. 05The Length at Which Everything Changesopening onlyPast a certain conversation length the model stops being the expensive thing to read, the conversation does, and the arithmetic of looking back stops being free.
  6. 06The One Trick That Actually Worksopening onlyReading the weights once and using them for many requests at the same time is the single change that turns a movement problem into an arithmetic one.
  7. 07Reading and Writing Are Different Jobsopening onlyTaking in a prompt and producing a reply obey opposite limits, want opposite settings, and interfere with each other when run on the same machine.
  8. 08What a Million Words Actually Costsopening onlyEverything in this course assembles into a single number, a price per million words, and that number ranks the available changes honestly.