Answers · The mathematics underneath

Why can't cross entropy loss go below the entropy of the labels?

In shortCross entropy cannot fall below the uncertainty already in the labels because it is that uncertainty plus a non-negative distance from the model to the labels.

It cannot because cross entropy is exactly the entropy of the label distribution plus a non-negative distance from the labels to the model. The entropy term is fixed by the labels before the model is built, so training can only reduce the distance term to zero, at which point the loss equals the entropy of the labels.

-\sum_i q_i\log p_i = \underbrace{-\sum_i q_i\log q_i}_{H(q)} + \underbrace{\sum_i q_i\log\frac{q_i}{p_i}}_{D(q\parallel p)}

Here $q_i$ is how often outcome $i$ actually occurs in the context, and $p_i$ is the probability the model gives it. The identity is obtained by adding and subtracting $\sum_i q_i\log q_i$ , which changes nothing. The first term, $H(q)$ , contains no $p$ at all. The second term, $D(q\parallel p)$ , contains the model and is zero only when the model matches the labels exactly.

The floor is a property of the labels

Take a context with two outcomes and $q=(0.5,0.5)$ , using natural logarithms. The label entropy is $-0.5\ln0.5-0.5\ln0.5=0.6931$ . A perfect model $p=(0.5,0.5)$ therefore reports a loss of $0.6931$ , not zero, and no model can beat it. A model $p=(0.7,0.3)$ reports $-0.5\ln0.7-0.5\ln0.3=0.7803$ , which is $0.7803-0.6931=0.0872$ above the floor. A model $p=(0.95,0.05)$ reports $-0.5\ln0.95-0.5\ln0.05=1.5235$ , which is $1.5235-0.6931=0.8304$ above the floor. The loss moves; the floor does not.

FIG 1

q first outcome	q second outcome	H(q)
1	0	0.000
0.5	0.5	0.693
0.9	0.1	0.325

The natural guess is that a lower loss always means a model closer to perfect, and that a perfect model would report zero. That is true only for one-hot labels. With $q=(1,0)$ the entropy is $-1\ln1-0\ln0=0$ . With $q=(0.5,0.5)$ the perfect model reports $0.6931$ . Reading that as error confuses two different quantities: the uncertainty already in the labels, and the model's distance from them. The distance is the only part that measures the model, and it is zero at $p=q$ .

FIG 2Four models against the same evenly split labels

step	p on the first outcome	loss	floor	distance above the floor	what happened
1	0.5	0.693	0.693	0	Exactly right. The loss equals the floor and the distance term is zero.
2	0.7	0.78	0.693	0.087	A modest overcommitment costs little.
3	0.95	1.524	0.693	0.83	Confident, and now paying for it.
4	0.01	2.308	0.693	1.614	Nearly ruled out something that happens half the time. This is the expensive mistake.

4 steps

The floor column is constant. The last column is the only one training can change.

Why the distance cannot be negative

The second term is a Kullback-Leibler divergence. It is zero when $p_i=q_i$ for every outcome, and positive otherwise. That is the statement that makes the floor a floor rather than a coincidence: every model's loss is $H(q)$ plus something non-negative.

\sum_i q_i\log\frac{q_i}{p_i}\ge 0,\quad \text{with equality only when }p_i=q_i\text{ for all }i.

The divergence is not symmetric. The loss uses $D(q\parallel p)$ , averaging over what actually happens. For $q=(0.5,0.5)$ and $p=(0.01,0.99)$ , the distance above the floor is $2.3076-0.6931=1.6145$ . The reverse divergence $D(p\parallel q)$ is $0.01\ln(0.01/0.5)+0.99\ln(0.99/0.5)=0.6371$ . The version used by the loss punishes a model far more harshly for putting near-zero probability on something that happens than the reverse would. That is why a model trained this way tends to hedge rather than commit.

The floor vanishes when the labels are one-hot. If $q=(1,0)$ , then $H(q)=0$ , and the loss is $-\ln p_1$ , which tends to zero as $p_1\to1$ . With hard labels, zero is a limit rather than a value attained by a finite model. If the labels are smoothed or genuinely noisy, the floor is positive again. If the model puts probability zero on an outcome with $q_i>0$ , the loss is infinite, not merely high. The floor is a property of the label distribution and the choice of loss, not of the optimiser.

Worth keeping

Cross entropy equals the entropy of the labels plus the Kullback-Leibler divergence from the labels to the model, so its minimum over all models is the label entropy.
A perfect model of a fifty-fifty label distribution still has loss ln 2 = 0.693, not zero.
The floor disappears only when the label distribution is one-hot, in which case the label entropy is zero.

Where this goes next

This question is one step of The Mathematics of Cross Entropy, a course that works through its subject from the beginning. The page that teaches this part of it is What Minimising It Is Really Doing, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course

Why can't cross entropy loss go below the entropy of the labels?

The floor is a property of the labels

Why the distance cannot be negative

Worth keeping

Where this goes next

Read alongside