Derive the loss almost every classifier and language model is trained on, from the probability of the data through to what its value does and does not promise, well enough to read a loss of 1.8 and say what the model believes

The Mathematics of Cross Entropy

A derivation-first account of the training loss itself, from likelihood and the logarithm through softmax, the gradient, the entropy floor, perplexity and calibration.

8 lessons, written and corrected before you arrived. Reading them here needs no account. The first reads the whole way through; the others open and then stop. Starting the course gives you your own copy, where every idea has problems standing under it and you can ask about any sentence.

Start reading

01The Number a Model Is Really Asked ForA classifier does not output an answer, it outputs a distribution, and training scores it on how much of that distribution landed on what actually happened.
02Three Reasons the Logarithm Is Not Optionalopening onlyTaking the log of the likelihood is usually presented as a convenience, and it is three separate things: a rescue from underflow, a decoupling of examples, and the only function that could do either.
03The Loss, Written Out in Fullopening onlyThe loss can now be assembled, and two features appear at once: the sum over classes has one live term, and the value it starts at is known in advance.
04Turning Scores Into a Distributionopening onlyA network's last layer outputs unconstrained numbers, and softmax is what two requirements leave once they have been imposed, with one property that makes the loss computable at all.
05The Simplest Gradient in the Subjectopening onlyDifferentiate this loss with respect to the scores and almost everything cancels, leaving the predicted distribution minus the observed one. That is why the two functions are paired.
06The Floor the Loss Cannot Go Belowopening onlySplit the loss into two pieces: the uncertainty already in the data, which no model can remove, and the distance from the model to the data, which is the only part training can reduce.
07What a Perplexity of Twelve Meansopening onlyPerplexity is the loss with the logarithm undone, which turns it into a count of effective choices. It is easy to read and easy to quote misleadingly, and both halves matter.
08When a Low Loss Still Lies to Youopening onlyThis loss is minimised by telling the truth, and yet trained networks are systematically overconfident. The reason is in the gradient, and there are two standard repairs.