Basis Lab

Answers · How models work

Why do we need a forward pass before we can backpropagate?

In shortThe gradient is built from local derivatives evaluated at the values the forward pass computed, so the loss alone cannot supply it.

A forward pass is needed because backpropagation is the chain rule evaluated at the values the forward pass produced, and those values are not recoverable from the loss alone. The loss is one number; the gradient has one component per weight, and each component depends on intermediate activations. That is a dimension mismatch, not a practical inconvenience.

∂L∂w=∂L∂y^ ∂y^∂w\frac{\partial L}{\partial w} = \frac{\partial L}{\partial \hat y}\,\frac{\partial \hat y}{\partial w}

Written out, the chain rule is a product of local derivatives. The backward pass starts with ∂L/∂y^\partial L/\partial \hat y, which is cheap and depends on the loss, then multiplies by ∂y^/∂w\partial \hat y/\partial w, which is the part that depends on the forward values. For a deep stack the product is longer, but every factor is a derivative of one operation evaluated at the inputs and outputs that operation saw during the forward pass. The loss gives the first factor only. The rest are facts about the computation, and they are recorded, or recomputed, from the forward pass.

The arithmetic that needs the forward values

Take the smallest model with one weight: prediction y^=wx\hat y = w x, squared loss L=(y^−y)2L=(\hat y-y)^2. Its gradient is dL/dw=2(y^−y)xdL/dw = 2(\hat y-y)x. Fix w=2w=2 and target y=1y=1, so the gradient is 2(2x−1)x2(2x-1)x. The table below uses the same weight and target, and therefore the same loss in both rows, but different inputs.

FIG 1
input prediction loss gradient

Why the natural guess fails

The natural guess is that the loss tells the weights how to move. It does not. The loss says how wrong the prediction is, not which weight made it wrong or in which direction. A single scalar cannot determine a vector with as many entries as the model has weights. In the table, loss 11 is consistent with gradient 00 at x=0x=0 and gradient 22 at x=1x=1. To tell those apart one needs the prediction y^\hat y and the input xx, both produced by the forward pass. The loss curve is a single height; the gradient is the slope in every coordinate. In a deeper model the same point is sharper: the gradient for a layer contains the activation of the layer before it, so the backward pass is reading values that only the forward pass can supply.

For a two-layer scalar model, h=w1xh=w_1x, y^=w2h\hat y=w_2h, and L=(y^−y)2L=(\hat y-y)^2. Backpropagation computes two gradients, and the forward activation hh appears in both.

∂L∂w2=2(y^−y)h,∂L∂w1=2(y^−y)w2x\frac{\partial L}{\partial w_2}=2(\hat y-y)h,\qquad \frac{\partial L}{\partial w_1}=2(\hat y-y)w_2x

The gradient for w2w_2 needs hh; the gradient for w1w_1 needs y^\hat y, hh and xx. Set w1=3w_1=3, x=2x=2, w2=0.5w_2=0.5 and y=1y=1. The forward pass gives h=3⋅2=6h=3\cdot2=6, y^=0.5⋅6=3\hat y=0.5\cdot6=3, and L=(3−1)2=4L=(3-1)^2=4. The gradients are dL/dw2=2(3−1)⋅6=24dL/dw_2=2(3-1)\cdot6=24 and dL/dw1=2(3−1)⋅0.5⋅2=2dL/dw_1=2(3-1)\cdot0.5\cdot2=2. The number 66 does not appear in the loss 44; it came from the forward pass and is still needed when the backward pass reaches w1w_1.

This is why the update is wt+1=wt−η∇wLw_{t+1}=w_t-\eta\nabla_w L. To evaluate ∇wL\nabla_w L at the current weights wtw_t, the backward pass must use the activations those weights produced. If the weights moved first, the gradient would describe a different function. Backpropagation is not guessing blame from the size of the loss. It is differentiating the same arithmetic that produced the loss, one local derivative at a time, and multiplying those local derivatives together.

The memory cost follows from this. The gradient is the same size as the model, and the activations that the backward pass needs are also kept for the duration of one step. That is why training needs several times the memory of running the model, and why the forward pass is not a disposable preview. It is the record of what the model did on this batch, and the backward pass is an audit of that record.

Where the dependency changes shape

Storing every activation is not the only option. Gradient checkpointing discards most activations and recomputes segments during the backward pass. That trades memory for extra forward computation, but it does not remove the dependency: a forward pass over those segments happens again. For a single linear layer with known inputs and targets, a hand-written gradient can be evaluated directly, but it still substitutes xx and ww into the formula, and those are forward values. The only case where a forward pass is genuinely unnecessary is when the gradient is already known from elsewhere, which is not what happens when a loss is used to train a model.

Worth keeping

  • A loss value is one number, while a gradient has one component per weight, so many different gradients can produce the same loss.
  • Backpropagation is the chain rule run backwards, and every link in that chain is evaluated at an activation from the forward pass.
  • If activations are not stored, they must be recomputed, which is another forward pass over the same examples.

Where this goes next

This question is one step of Training a Model From Scratch, a course that works through its subject from the beginning. The page that teaches this part of it is The Loop Everything Sits In, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. Why does a bigger batch make the gradient less noisy?Each step's gradient is an average, and an average's error shrinks with the square root of the number of examples, so four times the batch only halves the noise.
  2. Why train on a loss function instead of just using accuracy?Because accuracy is a pass or fail count that is flat almost everywhere, so it gives the optimiser no slope to follow, while a loss changes smoothly and tells every weight which way to move.
  3. Why did my loss suddenly spike and can I recover without restarting?Your loss spike is probably one bad batch, and you can usually recover by waiting or rolling back to a recent checkpoint instead of restarting the whole run.
  4. Why does training loss keep falling while validation loss starts rising?Training loss keeps falling because the model is memorising the batches it is shown; held-out loss rises when that memorisation stops helping on data it has not seen.
  5. Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
  6. Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.

Read alongside