The Honest State Of The Answer

Last timeThe Curve That Comes Back Down

Part of why very large models generalise is understood, part is conjecture with good evidence, and the remainder is genuinely open.

The previous lesson left a question rather than an answer. Past the threshold many

settings fit the training data exactly, the procedure picks a gentle one, and gentle

ones predict well. The first two parts of that sentence are solid. The third is where

the honest account gets uncomfortable, and this lesson is about separating what is

established from what is conjectured and what is simply unknown.

The choice nobody wrote down

Start with the part that is understood. When many settings achieve zero training

error, the training error cannot be what distinguishes them, so something else does,

and that something is the fitting procedure. It begins at small values and moves in

small steps, so it stops at the first adequate solution it reaches rather than the

largest, and in the simplest families it can be proved to arrive at the solution of

smallest size among all exact fits.

FIG 1How large the chosen solution is, as capacity grows

size of the gentlest available exact fit

This is the curve that makes the second descent intelligible. Adding capacity enlarges the collection of settings that fit the training data exactly, so the gentlest member of that collection gets gentler. The procedure is not being made more powerful as it moves right. It is being given more options, and the best of the options improves.

This is a penalty nobody added. It is not written into the objective, it cannot be

turned off, and it is doing a large part of the work usually credited to the

architecture. It also explains why the capacity lesson insisted that capacity belongs

to the procedure, since here the procedure is the only thing making a choice at all.

The lesson stops here

5 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

The Honest State Of The Answer

The choice nobody wrote down

5 more paragraphs to go

This is the reading half

Read alongside