Answers · How models work

Why train on a loss function instead of just using accuracy?

In shortBecause accuracy is a pass or fail count that is flat almost everywhere, so it gives the optimiser no slope to follow, while a loss changes smoothly and tells every weight which way to move.

Train on a loss because accuracy is a count of correct answers, and a count is flat almost everywhere: its derivative with respect to every weight is zero except at the exact points where a prediction flips. A loss is a smooth number that changes when the model becomes more or less confident, so it supplies a gradient for every weight on every step.

A count has no slope

Take one binary example, true label $y=1$ , model probability $p$ . Accuracy thresholds at $0.5$ : $A=1$ when $p>0.5$ and $A=0$ otherwise. Cross entropy is $L=-\ln p$ . At $p=0.9$ , $A=1$ and $L=-\ln 0.9=0.105$ . At $p=0.51$ , $A=1$ still, but $L=-\ln 0.51=0.673$ . The same accuracy covers a confident correct answer and a barely correct one, and the loss separates them by $0.568$ .

FIG 1

p	accuracy A	loss L	dL/dp
0.9	1	0.105	-1.111
0.51	1	0.673	-1.961
0.49	0	0.713	-2.041

Across a batch the same point holds. Take three examples with true labels $[1,0,1]$ and predicted probabilities $[0.9,0.2,0.6]$ . With a $0.5$ threshold every prediction is on the correct side, so accuracy is $3/3=1$ . The cross entropy is the average of $-\ln 0.9=0.105$ , $-\ln(1-0.2)=-\ln 0.8=0.223$ , and $-\ln 0.6=0.511$ , which is $(0.105+0.223+0.511)/3=0.280$ . Change the probabilities to $[0.51,0.49,0.51]$ and accuracy is still $3/3=1$ , but the loss is the average of three copies of $-\ln 0.51=0.673$ , so $0.673$ . The count cannot tell these two batches apart; the loss can, and its gradient points toward the first.

\frac{\partial A}{\partial w}=0\ \text{almost everywhere},\qquad \frac{\partial L}{\partial w}=p-y\ \text{for } L=-\ln p.

What the gradient sees

The optimiser is not reading the label or the thresholded answer. It is reading one scalar and asking, for each weight, how a small change would alter that scalar. For accuracy the answer is zero almost everywhere, because a small change in a weight changes a probability by a small amount without changing which side of $0.5$ it lies on. At the boundary the answer is a jump: the function is discontinuous, so the derivative is not defined in the ordinary sense. A gradient made of zeros and undefined spikes cannot train a model with many weights. Cross entropy gives a finite slope everywhere $p$ is not exactly $0$ or $1$ , and that slope is largest when the model is confidently wrong. With $p=\sigma(wx)$ and $x=1$ , the chain rule gives $\partial L/\partial w=p-y=-0.1$ at $p=0.9$ . The update $w_{t+1}=w_t-\eta\nabla_w L$ with $\eta=0.1$ moves $w$ by $-0.1\times(-0.1)=+0.01$ . The accuracy gradient is zero, so the same update would move nothing.

The natural guess is that if accuracy is what you care about, you should optimise accuracy directly. That confuses the score you report with the score the backward pass can differentiate. Accuracy also hides confidence: a prediction at $p=0.51$ and one at $p=0.9$ count the same if both are on the correct side, even though the second is much less likely to flip under a small weight change. A loss such as cross entropy keeps that confidence information, so the update can keep improving a correct answer instead of stopping as soon as it crosses the threshold. The loss is not a proxy for accuracy in the sense of being a convenient approximation to the same number; it is a different number, chosen because it has the slope that accuracy lacks.

Where accuracy still belongs

Accuracy remains the right number for reporting, for comparing finished runs, and often for early stopping when the classes are balanced. It is also the right objective if you abandon gradients and use a black-box search that only needs a score, though that costs many more evaluations. With gradient descent the objective has to be differentiable almost everywhere and has to change when the model becomes more confident. If the loss is badly chosen, the model will minimise that loss exactly: on a heavily imbalanced set, a model can reach high accuracy by always predicting the majority class while the loss and the behaviour you care about are poor. That is a reason to choose a loss that encodes the cost of errors, not a reason to train on the count.

Worth keeping

Accuracy is a step function of the weights, so its gradient is zero almost everywhere.
A loss is the only part of the training loop that says what good means.
Accuracy is for reporting and model selection; the gradient needs a smooth surrogate.

Where this goes next

This question is one step of Training a Model From Scratch, a course that works through its subject from the beginning. The page that teaches this part of it is The Loop Everything Sits In, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course