The Word That Settles The Argument
Last timeKeeping Some Back
Capacity is the variety of patterns a model could express, and the parameter count is a surprisingly poor way to measure it.
Two people can look at the same model and disagree about whether it is big. One
counts the adjustable quantities and says a billion, which sounds enormous. The
other points out that nearly all of those quantities are held close to zero by a
penalty and that training stopped after a fraction of a pass, so the thing is
behaving like something far smaller. Both are describing the same object. They are
measuring different things, and the word that covers the disagreement is capacity.
What is being counted
Capacity is a property of a family of patterns, assessed before any data arrives.
Fix the kind of model you intend to use, leave its adjustable quantities free, and
ask how varied the resulting collection of input-to-output relationships is. A
straight line through one input has two things you may set, so the family contains
every straight line and nothing else. A parabola has three. A large network has a
number of free quantities too large to think about and a correspondingly vast
collection of relationships available to it.
Notice what capacity is not. It is not performance: a family with enormous capacity
contains a correct pattern for almost any task and millions of ruinous ones too, and
its size says nothing about which the training procedure will land on. It is not
complexity in the colloquial sense either, since a lookup table with no adjustable
quantities has unlimited capacity in the sense that matters here.
What capacity does tell you is how seriously to take a training score. A family able
to express only straight lines cannot fit a hundred scattered points closely, so if
it does, something real has been found. A family able to express anything fits those
points closely whether or not there is anything in them to find, so a close fit is no
evidence. That is the hinge the whole subject turns on.
The lesson stops here
7 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents