Basis Lab

Answers · How models work

What is actually new in a neural network compared with linear regression?

In shortOnly three things are new: the slopes are computed by working backwards, the score is no longer a bowl, and the model builds its own inputs. The prediction, the loss, the step and the held out check are unchanged.

A neural network is a prediction, a score, a slope for every knob and a step against the slope, which is exactly the four parts a fitted straight line already has. Three things are genuinely new: the slopes are computed by working backwards through the layers, the score is no longer a bowl, and the inputs are built by the model rather than chosen by a person. Everything else is the same machinery at a larger size.

FIG 1One unit of a layer
the output of unit m, which becomes an input to the next layer
the knob unit m puts on input j, so a layer of M units over k inputs has M times k of them
a squashing function applied to the total
unit m's offset
Inside the bracket is the straight line; outside is the squash. A layer is M of these side by side, and each one's output becomes an input to the next layer.

The same machinery, counted

For one input, the line y^=wx+b\hat{y} = wx + b has 2+1=32+1 = 3 knobs. Three hidden units reading two inputs carry 3×2=63 \times 2 = 6 weights and 3 offsets, and one output unit reading those three adds 3 weights and 1 offset, so 13 knobs in all. Watch what happens to a knob, old or new. Take x=1x = 1 with target y=3y = 3, start at w=1w = 1, b=0b = 0, step size 0.10.1. The prediction is 1, the error y^−y\hat{y} - y is −2-2, the squared score is 4. The slope in ww is 2(y^−y)x=2(−2)(1)=−42(\hat{y} - y)x = 2(-2)(1) = -4, so ww becomes 1−0.1(−4)=1.41 - 0.1(-4) = 1.4. A second pass: prediction 1.4, error −1.6-1.6, score 2.56, slope 2(−1.6)(1)=−3.22(-1.6)(1) = -3.2, so ww becomes 1.4+0.32=1.721.4 + 0.32 = 1.72. A network runs that same sequence, with the prediction computed as one pass forward through the layers, the same squared error on top, and the same subtraction at the end.

The guess that fails is that a network is a different kind of object, so the shared parts go unrecognised. Linear regression is not a cousin of a network, it is the last layer of one, and everything in front of that layer exists to hand it a better set of inputs. The extra layers add repetitions, not a new ingredient, and the repetition only counts as depth because of the squash. Without it, two layers multiply out into one line.

w2(w1x+b1)+b2  =  (w2w1) x  +  (w2b1+b2)w_2\big(w_1 x + b_1\big) + b_2 \;=\; (w_2 w_1)\,x \;+\; (w_2 b_1 + b_2)

The three things that are new

With w1=2w_1 = 2, b1=1b_1 = 1, w2=3w_2 = 3, b2=−4b_2 = -4 that stack is 6x−16x - 1, a single line with two knobs' worth of freedom and no more, however many layers you add.

The first new thing is how the slopes are computed. A knob in an early layer reaches the score only through everything after it, so its slope is a product of one factor per layer crossed. Each factor is shared by every knob below it, and working backwards from the score computes each factor once instead of once per knob. The backward pass costs about twice a forward pass, so one fitting cycle costs three forward passes' worth of arithmetic: at 10910^9 knobs that is 3,000,000,000 multiply and add operations per cycle, and no more than that, which is why a model that size is fittable at all.

The second is the shape of the score. With a straight line and squared error the score is quadratic in the knobs and the bottom can be solved for in one line. Nesting multiplies knobs together, and a product has more than one bottom: (uv−3)2(uv - 3)^2 is zero at u=v=3≈1.732u = v = \sqrt{3} \approx 1.732 and again at u=v=−1.732u = v = -1.732. Nothing can be solved for, so stepping downhill is the only method, where the fit ends up depends on where it started, and the answer is a good setting rather than the best one. Starting every knob at zero is wrong in a specific way: with all weights zero every unit in a layer outputs σ(0)=1/2\sigma(0) = 1/2, so the units are identical, receive identical slopes, and stay identical at every step thereafter.

The third is that the inputs are learned, and it is the reason networks get used. Four measurements into a straight line is 4+1=54+1 = 5 knobs, and a person chose those four. A million pixels into a thousand units is 1,000,000×1000=1091{,}000{,}000 \times 1000 = 10^9 knobs, and not one of them usefully multiplies a single pixel on its own. The early layers build the description that the last layer reads, so what gets fitted is the relationship and the variables it is stated in, not just the relationship.

Where the equivalence runs out

The reduction is exact for prediction, loss, slope and step, and it stops being useful when depth buys nothing. On a table of a few thousand rows and four columns the hand chosen inputs are already the right ones, and a straight line with a penalty on the size of its knobs is often as good as anything deeper. The landscape point cuts both ways as well: with very many knobs the poor stopping points turn out to be rare, and most places the fitting settles score about as well as each other, so the lost guarantee costs less in practice than a picture of a lumpy surface suggests.

Worth keeping

  • A network's last layer is a linear model, so the prediction, the loss, the step against the slope and the held out check are the same four things a straight line uses.
  • Three features are genuinely new: slopes computed backwards through the layers, a score that is no longer a bowl, and inputs the model builds for itself.
  • Delete the squash between layers and any depth of stack collapses into one straight line, so the squash is what makes depth mean anything.

Where this goes next

This question is one step of The Simplest Model That Learns, a course that works through its subject from the beginning. The page that teaches this part of it is The Same Four Parts, Much Bigger, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. Why does logistic regression use a sigmoid instead of a straight line?The sigmoid turns an unbounded straight-line score into a probability between zero and one, and it pairs with the log score so fitting keeps the same gradient shape.
  2. What is the difference between the inputs and the weights in a linear model?The input is supplied by each example and changes from row to row; the weights are the model's adjustable numbers, chosen once by fitting and shared by every example.
  3. How do I know if I should increase or decrease the slope to reduce the error?The sign of how fast the loss changes with the slope tells you: positive means lower the slope, negative means raise it.
  4. Why does linear regression square the errors instead of just adding their absolute values?Squaring makes large errors count much more than small ones, so it picks a different line from absolute value; absolute value is a different choice about costs, not a mistake.
  5. Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
  6. Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.

Read alongside