Answers · The mathematics underneath

Why do machine learning models use log loss instead of just multiplying probabilities?

In shortMultiplying many probabilities underflows to zero on a computer; adding their logarithms keeps the same score and stays representable.

The score is kept as a sum of logarithms because the product it represents is too small for a computer to hold. A model's likelihood for a long document is a product of hundreds or thousands of probabilities, each below one, and that product falls below the smallest positive floating point number and becomes exactly nought; its logarithm is an ordinary negative number, so training and comparison can continue.

L=\prod_{i=1}^N p_i,\qquad \log L=\sum_{i=1}^N\log p_i.

Why the product dies

Take a document of 680 words, each assigned probability 0.3. The natural logarithm of 0.3 is -1.20397, so the log of the product is 680 times that, -818.70. Dividing by ln 10, which is 2.302585, gives -355.56, so the product itself is 10 to the minus 355.56. Double precision has a smallest positive value around 10 to the minus 308. The product is below it by nearly fifty orders of magnitude, so a loop that multiplies returns 0.0, not a small positive number. The sum -818.70 is an ordinary number and is carried at full precision. At 100 words the product is 10 to the minus 52.29 and survives; at 2000 words the sum is -2407.95 and the product would be 10 to the minus 1045.8, which cannot be written in the format at all.

The tempting inference is that multiplication is the definition, so it must be the right computation, and that a result of nought is merely a very small probability rounded down. It is not rounding. The floating point format stores a fixed number of significant digits and a separate exponent; that exponent has a floor. Once the product crosses the floor, the stored value is exactly nought and every digit of information is gone. Dividing by it fails, taking its logarithm fails, and comparing two such products finds them equal. A model comparison that reports two candidates as having probability nought looks like a tie and is an arithmetic failure, not a statement about the models.

What is being averaged

Log loss makes the score per example. If the model gives probability p_i to the correct outcome on example i, the log loss is minus the average log probability, not the average probability. For N independent examples the likelihood is the product of the p_i, so its logarithm is the sum of the log p_i. Because the logarithm is increasing, maximising that sum is the same as maximising the product. Negating and dividing by N turns it into a loss that smaller is better and whose size does not grow with the length of the document. This average negative log probability is cross entropy.

\mathrm{log\ loss}=-\frac1N\sum_{i=1}^N\log p_i.

Nothing important is lost by staying in log space. The logarithm preserves order, so comparing two probabilities is comparing their logs. A ratio p/q becomes log p minus log q. A product of independent probabilities becomes a sum of logs, which is the operation that started this. The right average when quantities multiply is the geometric mean, and its log is the arithmetic mean of the logs, so the per-word score is the correct summary rather than a convenient one. Perplexity is the exponential of that average, reported so the number reads as an effective number of choices.

The one hard case

Addition has no counterpart in log space. There is no rule that writes log(p+q) using only log p and log q, so combining alternatives means exponentiating first. That step is dangerous because it reverses the protection: a log probability of -800 exponentiates to 0.0, and a large positive value overflows to infinity. To add two such quantities you cannot simply exponentiate both at their stored size.

\log\sum_i e^{a_i}=m+\log\sum_i e^{a_i-m},\qquad m=\max_i a_i.

Subtract the largest log probability m before exponentiating. The largest term becomes e to the nought, which is 1, and every other term lies between 0 and 1. The sum is therefore between 1 and the number of terms, so it cannot overflow, and adding m back at the end restores the exact value. This is log-sum-exp. It is an identity, not an approximation; the only terms that underflow are those too small to have changed the sum. The same shift appears in softmax, for the same reason.

The raw product is harmless when N is small or the probabilities are not tiny; a product of ten probabilities near 0.3 is fine. Log loss matters when the product spans hundreds of orders of magnitude, as language models do. It also assumes every p_i is positive, since log 0 is undefined, so a model that assigns probability exactly nought to the correct outcome needs smoothing or clipping. And if you need actual probabilities rather than scores, exponentiate deliberately, preferably after subtracting a maximum, and expect the small ones to vanish.

Worth keeping

A product of hundreds of probabilities below one underflows to exactly zero in ordinary floating point, and the comparison then reports a tie.
A sum of logarithms carries the same likelihood in a range that ordinary floating point can hold.
Log loss is the average negative log probability, so minimising it is the same as maximising the product of the probabilities.

Where this goes next

This question is one step of Logarithms, Growth and Scale, a course that works through its subject from the beginning. The page that teaches this part of it is Why Every Score Is a Log Probability, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course