The Mathematics of Cross Entropy
Turning Scores Into a Distribution
Last timeThe Loss Itself
A network's last layer outputs unconstrained numbers, and softmax is what two requirements leave once they have been imposed, with one property that makes the loss computable at all.
The loss needs probabilities. A network's final linear layer produces whatever
real numbers the weights happen to give it: negative, large, unbounded. Those
numbers are called logits, or scores, and something has to stand between them
and the loss.
That something is not a design choice so much as a consequence of two
requirements.
Two requirements and one function
The output has to be positive, because a logarithm of a negative number is not
available, and it has to sum to one, because that is what makes it a
distribution. Neither can be asked of a linear layer: a layer with a constraint
on its outputs is a layer whose weights are no longer free.
So the constraints are imposed afterwards, by a fixed function with no
parameters. Making numbers positive is what an exponential does; making a
positive list sum to one is what dividing by the total does.
- the score, or logit, the network produced for class k
- the probability assigned to class k
- the number of classes
The usual objection is that other functions would also do. Squaring makes
numbers non-negative, and dividing by the sum of squares would normalise them.
It does not work: squaring collapses a score of and a score of into
the same number, so the sign of the evidence is destroyed.
The exponential has a property nothing else does. It turns addition into
multiplication, so adding two units to a score multiplies that class's odds
against every other class by a fixed factor, whatever the scores happened to be:
The lesson stops here
7 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents