The Floor the Loss Cannot Go Below

Last timeThe Gradient Is the Error

Split the loss into two pieces: the uncertainty already in the data, which no model can remove, and the distance from the model to the data, which is the only part training can reduce.

So far the loss has been a number to make small. This lesson asks a different

question: how small can it get, and what is actually being reduced on the way

down. The answer is that the loss has two parts, and training only touches one

of them.

Add and subtract one term

Write $q$ for the true probability of an outcome and $p$ for the model's. The

loss averaged over the data is $-\sum q \log p$ . Add and subtract

$\sum q \log q$ , which changes nothing:

FIG 1The loss, split into a part about the data and a part about the model

: how often outcome i actually occurs in this context
: the probability the model assigns to outcome i

An identity, not an approximation. The first term does not contain the model at all, so no amount of training can move it. The second is zero when the model matches the data and positive otherwise, so it is the entire object of the exercise.

Two consequences follow, and they are the whole lesson.

The lesson stops here

6 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

The Floor the Loss Cannot Go Below

Add and subtract one term

6 more paragraphs to go

This is the reading half

Read alongside