Answers · The mathematics underneath

Why does subtracting the max from the logits not change softmax?

In shortSubtracting the largest score is an exact rewriting of softmax, not a change to the distribution: the same factor cancels above and below.

Subtracting the maximum is an exact rewriting of softmax, not an approximation: every exponential is multiplied by the same factor, and that factor cancels between numerator and denominator. The probabilities are unchanged. What changes is only the size of the intermediate numbers, which is why implementations do it.

p_k = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}} = \frac{e^{z_k-m}}{\sum_{j=1}^{K} e^{z_j-m}} = \frac{e^{-m} e^{z_k}}{e^{-m} \sum_{j=1}^{K} e^{z_j}} = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}}

The cancellation

Write the shifted score as $z_j-m$ . The numerator for class $k$ becomes $e^{z_k-m}=e^{-m}e^{z_k}$ . Every term in the denominator becomes $e^{z_j-m}=e^{-m}e^{z_j}$ , so the denominator is $e^{-m}\sum_j e^{z_j}$ . Dividing gives the chain above. The factor $e^{-m}$ appears once on top and once on the bottom, so it cancels. Nothing else is left to change. This is an identity for every real $m$ and every finite set of scores: softmax has no distinguished origin on the score axis.

Take scores $2,1,0$ . The unshifted numerator for the first class is $e^2=7.389$ , and the denominator is $7.389+2.718+1=11.107$ , so $p=7.389/11.107=0.665$ . Subtract the maximum $2$ : the scores become $0,-1,-2$ , the exponentials become $1,0.3679,0.1353$ , the denominator is $1.5032$ , and $p=1/1.5032=0.665$ . The second class gives $2.718/11.107=0.245$ unshifted and $0.3679/1.5032=0.245$ shifted; the third gives $1/11.107=0.090$ and $0.1353/1.5032=0.090$ . The last digits differ only because the decimals are rounded.

\frac{p_a}{p_b} = \frac{e^{z_a}}{e^{z_b}} = e^{z_a-z_b} = e^{(z_a-m)-(z_b-m)}

Equivalently, the probability vector depends only on the differences $z_j-z_k$ . Add the same constant to every score and each difference is unchanged; subtract the same constant and each difference is unchanged again. The common part of the scores is a direction softmax cannot see. That is why a network can let the absolute level of its scores drift without changing its predictions, and why the final layer has one redundant direction in parameter space. It also tells you what the max subtraction is not: it is not a normalisation step that alters the model's confidence, because confidence is carried by the gaps between scores, not by their common level.

The natural guess is that subtracting a number from every score must lower each probability or change their ratios. It cannot change a ratio: the displayed identity says the odds of class $a$ against class $b$ depend only on $z_a-z_b$ , and a common shift leaves that difference unchanged. If every pairwise ratio is unchanged, the whole distribution is unchanged. The subtraction is not a modelling choice and it does not alter what the model believes. It is a numerical rewrite. The maximum is convenient, not essential; any constant would preserve the distribution.

Why the maximum

Any constant works in exact arithmetic, but the maximum has a special numerical property. Let $m=\max_j z_j$ . Then $z_j-m\le 0$ for every $j$ , so $e^{z_j-m}\le 1$ , and the largest term is exactly $e^0=1$ . With $K$ classes the denominator is a sum of $K$ terms each at most $1$ , so the denominator lies between $1$ and $K$ . No exponential in the calculation can overflow. Scores like $102,101,100$ have the same probabilities as $2,1,0$ , but a naive softmax computes $e^{102}$ , which overflows in single precision. The shifted calculation computes $1,e^{-1},e^{-2}$ instead.

m = \max_j z_j, \qquad z_j-m \le 0, \qquad e^{z_j-m} \le 1, \qquad 1 \le \sum_{j=1}^{K} e^{z_j-m} \le K

Where exactness stops

In exact arithmetic the identity holds for any finite $m$ . In floating point, the shifted and unshifted versions can disagree in the last digits because each exponential is rounded separately; that is rounding error, not a change in the distribution. A bad shift can still fail: if $m$ is smaller than the largest score, some exponent remains positive and may overflow, so the useful choice is the actual maximum. At the other end, very negative shifted scores underflow to zero, which gives those classes probability zero in the computation; their true probabilities are below the smallest representable positive number, so the effect on the returned distribution is negligible. The operation is safe because the large common factor is removed before it is ever formed, not because a new distribution is being defined.

The same cancellation is what makes cross entropy over softmax stable. Since $\log \sum_j e^{z_j} = m + \log \sum_j e^{z_j-m}$ , shifting every score by $m$ and adding $m$ back leaves the loss unchanged; computed together with the true-class score, the $m$ cancels. The loss becomes a log of a sum of exponentials of non-positive numbers minus a shifted score, so no division by a tiny number and no logarithm of zero appears. That is the practical payoff of the invariance: the distribution is mathematically the same, and the arithmetic no longer has to handle exponentials of enormous scores.

Worth keeping

Subtracting the largest score leaves softmax unchanged because the same factor multiplies every exponential and cancels.
The maximum is chosen only so the largest exponential becomes one and overflow is avoided.
Any constant shift would preserve the distribution, so only differences between scores affect the probabilities.

Where this goes next

This question is one step of The Mathematics of Cross Entropy, a course that works through its subject from the beginning. The page that teaches this part of it is Where the Probabilities Come From, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course