The Simplest Gradient in the Subject

Last timeWhere the Probabilities Come From

Differentiate this loss with respect to the scores and almost everything cancels, leaving the predicted distribution minus the observed one. That is why the two functions are paired.

The previous lesson left the loss in a form with no division and no fraction

inside a logarithm:

J = \log\!\Big(\sum_{j} e^{z_j}\Big) - z_t

Now differentiate it with respect to one of the scores. This is the calculation

the backward pass performs, and the result is the reason this loss is used.

The one piece of calculus

The second term is easy: $z_t$ depends on $z_k$ only when $k$ is the observed

class, so its derivative is $1$ in that case and $0$ otherwise.

The first term requires the chain rule once. The derivative of a logarithm is

one over its argument, and the derivative of the sum of exponentials with

respect to $z_k$ is $e^{z_k}$ , since every other term is a constant:

The lesson stops here

7 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

The Simplest Gradient in the Subject

The one piece of calculus

7 more paragraphs to go

This is the reading half

Read alongside