Basis Lab

Answers · How models work

How do I know if I should increase or decrease the slope to reduce the error?

In shortThe sign of how fast the loss changes with the slope tells you: positive means lower the slope, negative means raise it.

Work out the derivative of the loss with respect to the slope. A positive derivative means the loss rises when the slope rises, so decrease the slope; a negative derivative means the loss falls when the slope rises, so increase it.

∂L∂w=2n∑i=1n(y^i−yi) xi\frac{\partial L}{\partial w} = \frac{2}{n}\sum_{i=1}^{n} \big(\hat{y}_i - y_i\big)\,x_i

Reading the sign

The derivative is a rate: how much the loss changes per unit increase in the slope knob, with the offset held still. If ∂L/∂w\partial L/\partial w is positive, a small increase in ww raises LL, so the way down is to decrease ww. If it is negative, a small increase in ww lowers LL, so the way down is to increase ww. The step writes this once: w←w−η ∂L/∂ww \leftarrow w - \eta\,\partial L/\partial w. The minus sign turns the derivative into a direction. Do not confuse the fitted line's slope with the slope of the loss; the first is ww, the second is ∂L/∂w\partial L/\partial w.

w←w−η ∂L∂ww \leftarrow w - \eta\,\frac{\partial L}{\partial w}

For the six houses, take the starting point w=2w=2 and b=120b=120. The predictions y^=2x+120\hat{y}=2x+120 and the errors y^−y\hat{y}-y are below. The final column is what the slope derivative averages.

FIG 1
xypredictionerrorerror times x
52196224281456
6123024212732
752652705375
80290280-10-800
94330308-22-2068
110385340-45-4950
sum-32-5255

The sum of the final column is 1456+732+375−800−2068−4950=−52551456+732+375-800-2068-4950=-5255. That is not the same as the sum of the errors, which is −32-32. The slope derivative weights an error on a large house more heavily than an error on a small one, because the slope is multiplied by the input. A +28+28 error at x=52x=52 contributes 14561456, while a −45-45 error at x=110x=110 contributes −4950-4950, and the large-house underprediction wins.

∂L∂w=26(−5255)=−1751.67\frac{\partial L}{\partial w} = \frac{2}{6}(-5255) = -1751.67

The natural guess

The natural guess is to try increasing the slope, recompute, and keep the change if the loss falls, otherwise try the other way. That works, but it is trial and error, and it hides why the direction is what it is. The derivative is that trial done once in algebra: it gives the direction before any step is taken. The other common mistake is to look only at the average error. Here the average error is −32/6=−5.33-32/6=-5.33, which says the predictions are low on average; that is a statement about the offset, not about the slope. For the slope, an underprediction on a large input counts more than an overprediction on a small one. The sign of ∂L/∂w\partial L/\partial w, not the sign of the average error, chooses the direction for ww.

With a step size η=0.00001\eta=0.00001, the slope step is w←2−0.00001(−1751.67)=2.0175w \leftarrow 2 - 0.00001(-1751.67)=2.0175. The offset derivative is 2(−32)/6=−10.672(-32)/6=-10.67, so the offset step is b←120−0.00001(−10.67)=120.0001b \leftarrow 120 - 0.00001(-10.67)=120.0001. Both use the old setting. The original squared errors sum to 784+144+25+100+484+2025=3562784+144+25+100+484+2025=3562, so the original score is 3562/6=593.73562/6=593.7. After the step the squared errors sum to about 33903390, so the score is about 3390/6=565.03390/6=565.0. The score fell, as the negative derivative said it would.

Where the sign stops

The sign is only first order. If ∂L/∂w=0\partial L/\partial w=0, increasing or decreasing the slope makes no first-order difference; for this squared loss that coordinate is flat at the bottom, but with the offset also moving you are at a stationary point only in that direction. If the step size is too large, the derivative still points downhill, yet the step can cross the bottom and land higher, so the score rises even though the direction was right. The derivative does not choose η\eta. Rescaling the inputs also changes the size of the derivative, so the direction may stay the same while the step needs changing.

Worth keeping

  • A positive derivative of the loss with respect to the slope means increasing the slope would increase the loss, so the slope should go down.
  • The slope derivative weights each error by its input, so an error on a large input matters more than the same error on a small input.
  • The derivative gives direction only; the step size decides whether the loss actually falls.

Where this goes next

This question is one step of The Simplest Model That Learns, a course that works through its subject from the beginning. The page that teaches this part of it is Which Way to Turn the Knobs, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. What is actually new in a neural network compared with linear regression?Only three things are new: the slopes are computed by working backwards, the score is no longer a bowl, and the model builds its own inputs. The prediction, the loss, the step and the held out check are unchanged.
  2. Why does logistic regression use a sigmoid instead of a straight line?The sigmoid turns an unbounded straight-line score into a probability between zero and one, and it pairs with the log score so fitting keeps the same gradient shape.
  3. What is the difference between the inputs and the weights in a linear model?The input is supplied by each example and changes from row to row; the weights are the model's adjustable numbers, chosen once by fitting and shared by every example.
  4. Why does linear regression square the errors instead of just adding their absolute values?Squaring makes large errors count much more than small ones, so it picks a different line from absolute value; absolute value is a different choice about costs, not a mistake.
  5. Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
  6. Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.

Read alongside