Where to start

Reading tracks

The shelves are subjects. These are results. Each track is an order through courses that already exist, named after what you can do when you reach the end of it, because that is the thing a reader can actually recognise about material they have not met yet.

Nothing here is locked and nothing is a prerequisite. The order is the order I would read them in; starting at the fifth course of a track is allowed and sometimes right. Every course is also still on its shelf, and the library has more courses than appear here.

Track 1 · 8 courses

Read any page here without stopping at the symbols

For the reader who keeps bouncing off a formula and assumes the problem is them. Everything else in the library gets easier after this.

01
Reading the Notation8 lessons
The thirty marks every other course uses silently, each one named and said out loud.
02
Logarithms, Growth and Scale8 lessons
Nearly every score in the model courses is the log of something, and they all assume you know that.
03
Counting Without Listing8 lessons
The denominator sitting under every probability the library teaches.
04
Chance, Spread and What to Expect8 lessons
The small set of ideas every later course quietly assumes, now that you can count the cases.
05
Arguing That It Always Holds8 lessons
The argument shapes that every derivation in the library expects you to accept without being shown.
06
Vectors, Distance and Similarity8 lessons
What it means for two things to be close, and what goes wrong with that in many dimensions.
07
Matrices as Maps8 lessons
A matrix read as a thing that moves vectors rather than a grid of numbers.
08
Change and Accumulation8 lessons
How fast a rule changes and how much it adds up to, which turn out to be one question.

At the end You can open any course on any shelf and the mathematics is no longer the thing standing in your way.

Track 2 · 8 courses

Tell in advance which approach survives a million rows

The algorithms shelf, then the three courses about the machine underneath it. Needs nothing from the first track except its opening course.

01
How Long It Takes8 lessons
Counting the work instead of timing it, and telling a slow approach apart from a slow machine.
02
Finding It Instantly8 lessons
Why a lookup answers in the same time at ten items and at ten million, and what it can never do.
03
Keeping Things in Order8 lessons
Picks up exactly where the previous one stops, and the same structure turns up under a database index.
04
Answers Built From Smaller Answers8 lessons
Recursion, and the repair that turns an impossible search into a table you fill once.
05
Graphs and Finding a Path8 lessons
Things and the connections between them, and the few walks that answer most real questions.
06
Memory, All the Way Down8 lessons
Where the counting stops predicting anything, because two loops doing the same arithmetic differ a hundredfold.
07
What Makes Code Fast8 lessons
Speed predicted from what the hardware has to do rather than from how many operations there are.
08
Reading a Profile Honestly8 lessons
Measuring where the time went and believing the answer, which closes the loop on the whole track.

At the end You can choose between two approaches before writing either one, and prove afterwards that the change helped.

Track 3 · 8 courses

Build a model by hand, then find the same parts inside every large one

Starts with a model small enough to do on paper and ends with nothing in a network left unexplained. Wants the first track first, or at least its opening two courses.

01
The Simplest Model That Learns8 lessons
A prediction, a loss, a gradient and a step, done entirely by hand. Every later model is this.
02
Why Learning Works At All8 lessons
Why a model that fits its training data may still be useless, before you build anything bigger.
03
The Mathematics of Gradient Descent8 lessons
Why the step works, what breaks it, and what every optimiser since has been repairing.
04
The Mathematics of Cross Entropy8 lessons
The loss almost everything is trained on, to the point of reading a value of 1.8 and saying what the model believes.
05
Backpropagation in Full8 lessons
The gradient followed from the loss back to every weight, and what that costs in memory.
06
Keeping The Numbers In Range8 lessons
Why deep networks fall apart numerically, and what initialisation and normalisation each do about it.
07
Trees, and Committees of Them8 lessons
The family that actually wins on table-shaped data, which is most data anybody has.
08
Finding Structure Without Labels8 lessons
What to do when nothing is labelled, and how to judge the result with no right answer to check against.

At the end You can read a training curve and say which number to change, and no part of a model is a black box to you.

Track 4 · 8 courses

Explain the model everybody is actually using

The language model, followed from a string of characters to a set of weights and then to something small enough to serve. Wants the model track first.

01
How Text Becomes Numbers8 lessons
Every step between characters and the vectors a model consumes.
02
Cutting Text Into Pieces8 lessons
Building the vocabulary rather than downloading one, which explains most surprising costs and failures.
03
The Mathematics of Attention8 lessons
Derived rather than described, to the point of reading the equations in a paper without skipping them.
04
How a Model Picks Its Next Word8 lessons
What every sampling setting does to the distribution, and how to tell a model failure from a decoder failure.
05
Where Training Data Comes From8 lessons
Judging a training set, which is where most disappointing results are decided.
06
Training a Model From Scratch8 lessons
One model from a pile of text to working weights, with every decision on the way named and priced.
07
Adapting a Model You Did Not Train8 lessons
What a small change actually alters, what adaptation destroys, and when not to bother.
08
Making a Model Smaller8 lessons
Predicting the memory, the speed and the quality each technique gives you before running it.

At the end You can read a paper's method section or a model card straight through, and say what it leaves out.

Track 5 · 9 courses

Put a model in front of people and know whether it is working

The longest of the five, at nine courses, because measuring is four courses on its own and shortening it would be the dishonest cut.

01
Writing What the Model Reads8 lessons
The input as a thing you design to a budget, not a thing you type.
02
Finding Things by Meaning8 lessons
Retrieval end to end, since most of what goes into that input was fetched.
03
Models That Use Tools8 lessons
The loop where a model does things rather than only says them, and which failures belong to which part of it.
04
When a Model Is Sure and Wrong8 lessons
Reading the number beside an answer for what it is, which decides when a system may act alone.
05
What a Model Refuses, and Why8 lessons
The layer that decides what the system will not do, and what each defence costs in wrongly blocked requests.
06
Reading a Benchmark Honestly8 lessons
How much of a reported score is real, before you start reporting your own.
07
Judging a System, Not a Model8 lessons
An evaluation for the product rather than the model, and saying what your numbers do not cover.
08
Running an Online Experiment8 lessons
Whether the change helped, including the several ways an experiment reports a win that is not there.
09
Watching a Model After Launch8 lessons
The months after the day it shipped, measured without labels.

At the end You can say whether your change helped, and catch a regression before a user reports it.