Basis Lab

Answers · The mathematics underneath

Why does cross entropy loss only use the probability of the correct class?

In shortBecause a definite label puts all the target weight on the correct class and none on the others, so every other term is multiplied by zero.

The loss only uses the probability of the correct class because a definite label is a one-hot vector: it puts 1 on the observed class and 0 on every other. In the sum over classes every term except the true one is multiplied by 0, so the live term is the negative log of the probability assigned to the correct class.

J=−1N∑i=1N∑k=1Ktiklog⁡pikJ = -\frac{1}{N}\sum_{i=1}^{N}\sum_{k=1}^{K} t_{ik}\log p_{ik}

For a single example the outer average is irrelevant. Take three classes, an observed class 2, target (0,1,0)(0,1,0) and model (0.25,0.60,0.15)(0.25,0.60,0.15). The inner sum has three terms, but two of them carry a target of zero. Using natural logarithms, the loss on that example is

Ji=−(0log⁡0.25+1log⁡0.60+0log⁡0.15)=−log⁡0.60=0.5108J_i = -\big(0\log 0.25 + 1\log 0.60 + 0\log 0.15\big) = -\log 0.60 = 0.5108
FIG 1
classtarget tmodel pt x (-log p)
100.250
210.600.5108
300.150

The zeros come from multiplication, not from the wrong-class probabilities being small. The model could have put 0.25 and 0.15 elsewhere, in any split that leaves the true class at 0.60, and the loss on this example would not move. The value 0.5108 is the entire loss: it is −log⁡0.60-\log 0.60, since log⁡0.60=−0.5108\log 0.60=-0.5108.

The common misreading is that because the formula sums over all KK classes, every predicted class probability contributes to the loss. It does not. The target vector decides which terms survive, and for a hard label all but one of its entries are exactly zero. The wrong-class probabilities are present in the expression, but they are multiplied by zero, not added with small weights. That is why an implementation can take a vector of predictions and an integer label: the integer is the one-hot target with the zeros omitted. A sum over all classes is still the right general form because it also covers soft targets; it is not a claim that all classes matter for a hard label.

There is a real consequence. Suppose the true class gets 0.60. Model A gives (0.60,0.35,0.05)(0.60,0.35,0.05) and model B gives (0.60,0.05,0.35)(0.60,0.05,0.35). Both have loss −log⁡0.60=0.5108-\log 0.60=0.5108. The loss cannot distinguish a near miss from an absurd one, because it never reads the 0.35 or the 0.05. If that distinction matters, it has to enter through the targets or through another term.

When the target is not one-hot

The collapse depends on the target being certain. With label smoothing, distillation, or genuinely uncertain labels, several entries of tt are nonzero and every corresponding class contributes. For target (0.7,0.2,0.1)(0.7,0.2,0.1) and model (0.5,0.3,0.2)(0.5,0.3,0.2) the loss is

Ji=−(0.7log⁡0.5+0.2log⁡0.3+0.1log⁡0.2)=−(0.7(−0.6931)+0.2(−1.2040)+0.1(−1.6094))=0.8869J_i = -\big(0.7\log 0.5 + 0.2\log 0.3 + 0.1\log 0.2\big) = -\big(0.7(-0.6931)+0.2(-1.2040)+0.1(-1.6094)\big) = 0.8869

All three terms are live there, and the loss now depends on how the model distributed probability over the wrong classes. The one-live-term form is the special case, not the definition.

Two classes and the uniform start

For two classes the general expression is still the loss. Write the model's two numbers as pp and 1−p1-p, and the target's as tt and 1−t1-t. Then

Ji=−(tlog⁡p+(1−t)log⁡(1−p))J_i = -\big(t\log p + (1-t)\log(1-p)\big)

If t=1t=1, the second term vanishes and Ji=−log⁡pJ_i=-\log p. If t=0t=0, the first term vanishes and Ji=−log⁡(1−p)J_i=-\log(1-p). Binary cross entropy is not a different loss; it is the same sum with two classes and the second probability written as one minus the first.

Worth keeping

  • With a one-hot target, cross entropy on an example collapses to the negative log of the probability assigned to the true class.
  • The loss is indifferent to how probability is spread over the wrong classes when the label is certain.
  • An untrained uniform model over K classes has loss log K, so a ten-class run should begin near 2.3026.

Where this goes next

This question is one step of The Mathematics of Cross Entropy, a course that works through its subject from the beginning. The page that teaches this part of it is The Loss Itself, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. Why can't cross entropy loss go below the entropy of the labels?Cross entropy cannot fall below the uncertainty already in the labels because it is that uncertainty plus a non-negative distance from the model to the labels.
  2. What does a perplexity of 10 actually mean?A perplexity of 10 means the model's average loss is about 2.30 nats per prediction, equivalent to a uniform choice among ten equally likely options, not a 10 percent accuracy.
  3. What does temperature scaling actually change in a model's probabilities?It divides every score by one positive number before the softmax, so the probabilities become sharper or flatter while the ranking and top prediction stay the same.
  4. Why does subtracting the max from the logits not change softmax?Subtracting the largest score is an exact rewriting of softmax, not a change to the distribution: the same factor cancels above and below.
  5. What is the difference between variance and standard deviation?They measure the same spread: variance is the average squared distance from the mean, standard deviation is its square root, so you report the standard deviation and calculate with the variance.
  6. How do you make a vague claim precise enough to tell whether it's true?Fix what is being talked about, what is asserted, which cases are covered and the cut-off, then name an observation that would prove it false.

Read alongside