The Mathematics of Cross Entropy
Three Reasons the Logarithm Is Not Optional
Last timeWhat the Loss Is Measuring
Taking the log of the likelihood is usually presented as a convenience, and it is three separate things: a rescue from underflow, a decoupling of examples, and the only function that could do either.
The previous lesson ended with a product of probabilities to be maximised.
The standard next move is to take its logarithm, usually introduced with the
words "for convenience". That undersells it considerably. There are three
separate things the logarithm does here, and only one of them is convenience.
Start with the reason the move is allowed at all.
The move is free
The logarithm is strictly increasing: if then . So the
parameters that make the likelihood largest are exactly the parameters that make
its logarithm largest. Not approximately, and not usually. The set of maximisers
is identical.
This is worth saying plainly because it means everything that follows costs
nothing. We are not trading accuracy for tractability. We are relabelling the
vertical axis of the same landscape, and the location of the summit does not
care about the labels.
By convention the sign is then flipped so that we minimise rather than maximise,
and the whole thing is divided by :
- the probability the model gave to the label observed on example i
- the loss, which is what a training log prints
- the number of examples being averaged over, usually a batch
That expression is the negative log likelihood, and for a classifier it is what
everybody calls the cross entropy loss. The next lesson shows those are the same
thing. First, why the logarithm had to be there.
The lesson stops here
9 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents