Answers · How models work

Why is the model's noise prediction really the score of the noised image distribution?

In shortThe network's noise prediction, divided by the noise level and given a minus sign, is the direction that makes a noised image more plausible.

The prediction is the score because the forward process is Gaussian, so the noise and the gradient of the log density are related by one fixed division and one sign. If the predicted noise is $\varepsilon_\theta(x,t)$ and the noise level is $\sqrt{1-\bar\alpha_t}$ , the score is $-\varepsilon_\theta(x,t)/\sqrt{1-\bar\alpha_t}$ .

FIG 1The identity

: the distribution of real pictures blurred by the noise belonging to time t
: the direction from this point that most increases plausibility, with length saying how steeply
: the trained prediction of the noise present in this input
: the noise level at this time, which is the only thing converting between the two

The equality is exact for an optimal noise predictor. The minus sign says that removing the predicted noise and moving uphill in log density are the same direction.

Where the divisor comes from

Take one clean image $x_0$ and the usual variance-preserving step. The noised point is $x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\epsilon$ with $\epsilon\sim N(0,I)$ . Conditioned on $x_0$ , the density of $x_t$ is Gaussian with mean $\sqrt{\bar\alpha_t}x_0$ and variance $(1-\bar\alpha_t)$ in every direction. Differentiating its log with respect to $x_t$ gives minus the displacement from that mean, divided by the variance. But the noise that was added is exactly $\epsilon=(x_t-\sqrt{\bar\alpha_t}x_0)/\sqrt{1-\bar\alpha_t}$ , so the same derivative is $-\epsilon/\sqrt{1-\bar\alpha_t}$ . For a whole distribution rather than one image, the optimal network output is the conditional mean of $\epsilon$ given $x_t$ , and the identity carries through with that mean.

p_t(x_t\mid x_0)=N\!\left(\sqrt{\bar\alpha_t}x_0,\,(1-\bar\alpha_t)I\right),\qquad \nabla_{x_t}\log p_t(x_t\mid x_0)=-\frac{x_t-\sqrt{\bar\alpha_t}x_0}{1-\bar\alpha_t}=-\frac{\epsilon}{\sqrt{1-\bar\alpha_t}}.

The missing divisor

The common slip is to treat the predicted noise as the score itself, up to a sign. The divisor is not bookkeeping: it changes with the noise level and it changes the size of the field. Suppose the network predicts the same value $\epsilon_\theta=0.3$ at two times. At $\bar\alpha_t=0.96$ the divisor is $\sqrt{0.04}=0.2$ , so the score is $-0.3/0.2=-1.5$ . At $\bar\alpha_t=0.36$ the divisor is $\sqrt{0.64}=0.8$ , so the score is $-0.3/0.8=-0.375$ . The same prediction is four times stronger as a gradient at the lower noise level. Forgetting the divisor erases the fact that the field is steep near the data and shallow far from it, and that is exactly the variation the sampler uses as it reduces the noise.

FIG 2

alpha bar	noise level	predicted noise	score
0.96	0.2	0.3	-1.5
0.75	0.5	0.3	-0.6
0.36	0.8	0.3	-0.375

Why the blur is not cosmetic

A clean photograph is a point in a space of about three million dimensions, and real photographs fill a region so thin that the density is zero almost everywhere else. Where the density is zero its gradient is zero too, and zero is not a small amount of guidance: it is no direction at all. Blurring with noise spreads the density over the space, so a point away from the data still has a slope telling it which way the data lies. The price is that the blurred density no longer distinguishes the fine detail of the original, which is why the noise level is reduced as the walk proceeds. The log also removes the normalising constant: multiplying a density by any constant leaves its log gradient unchanged, so the one part of the density that cannot be computed from samples never appears in the score.

Where the identity stops

The identity is exact only under the assumptions that produced it. At $\bar\alpha_t=1$ the divisor is zero and there is no score to speak of, which is why the path never quite reaches a clean image in the training schedule. If the forward process is not the variance-preserving Gaussian one, the divisor is replaced by whatever the corresponding noise level is, and the sign and scale must be rederived. If the network predicts $x_0$ or a velocity rather than the noise, the right hand side must be converted before it is a score. And for a network that has not converged, the equality is an approximation: the optimal predictor gives the score, the trained one gives an estimate of it.

Once the output is read as a field, the denoising loop is only one way to follow it. The field says which way to move and how strongly at every point; a sampler can follow it with Euler steps, with a higher-order method, or with anything else that advances along a direction. That is why the same trained network can be used in samplers that look nothing like the original loop: they are changing the numerical method, not the quantity the network computes.

Worth keeping

A perfect noise predictor and the gradient of the log density differ only by division by the noise level and a sign.
Blurring with noise is what gives the density a usable slope away from the data.
The normalising constant of the density never enters the score, which is why the score can be learned from samples.

Where this goes next

This question is one step of How Noise Becomes a Picture, a course that works through its subject from the beginning. The page that teaches this part of it is What the Prediction Really Is, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course