The Simplest Model That Learns
The Same Four Parts, Much Bigger
Last timeWhen It Learns the Noise
A neural network is a prediction, a score, a gradient and a step, which are the four parts of this course, and the three things that are genuinely new are worth naming exactly.
The promise, made good
The first lesson claimed that a straight line contains every part a neural
network has. This lesson shows where each part is.
A layer of a network is several copies of the model from this course, side by
side. Each one takes all the inputs, has its own knobs, and produces one
number. Ten copies produce ten numbers. Those ten numbers are then squashed,
exactly as in the sixth lesson, and handed to the next layer as its inputs.
- the output of unit m in this layer, which becomes an input to the next
- unit m knob for input j, so a layer of M units over k inputs has M times k of them
- a squashing function, the same role as in the sixth lesson
- unit m offset
The squashing is not decoration. A straight line of a straight line is a
straight line, so a network without squashing between its layers, however
deep, collapses into one straight line with one layer's worth of expressive
power. The squash is the only thing that makes depth mean anything, which is
why it had to be introduced before this lesson rather than inside it.
| present in a network | changed from this course | which lesson it came fro | |
|---|---|---|---|
| a prediction with knobs | 1 | 1 | 1 |
| a score saying how wrong | 1 | 0 | 2 |
| a slope for each knob | 1 | 1 | 3 |
| a step against the slope | 1 | 0 | 3 |
| a penalty on the knobs | 1 | 0 | 7 |
| held back data | 1 | 0 | 7 |
What is new: slopes computed backwards
A network has a great many knobs, and every one of them needs a slope. The
chain from a knob in an early layer to the final score runs through everything
after it, so the slope is a product of many factors.
Computing each knob's slope separately would recompute the same shared factors
over and over. Computing them from the output end backwards computes each
shared factor once and reuses it for every knob that depends on it. That is
the entire idea, and it is the same idea as the sixth lesson of a course on
recursion: notice the overlap, and keep each answer instead of recomputing it.
The lesson stops here
4 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents