Answers · How models work

How do you turn the predicted noise into the mean of the previous image?

In shortSubtract a small scheduled fraction of the predicted noise from the current image, then rescale to undo the forward step's shrinkage.

The mean of the previous image is $x_t$ with a small scheduled fraction of the predicted noise removed, then divided by $\sqrt{\alpha_t}$ . The network does not hand you the mean directly; it hands you $\varepsilon_\theta(x_t,t)$ , and the schedule supplies the two coefficients that turn that guess into the mean.

\mu_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\,\varepsilon_\theta(x_t,t)\right), \qquad x_{t-1}=\mu_{t-1}+\sigma_t z.

Where the conditioning goes

Conditioning backwards is not the forward step run in reverse. The forward step $q(x_t\mid x_{t-1})$ tells you how to add noise to one particular previous image. Once you condition on a particular $x_t$ , many previous images could have produced it, so $q(x_{t-1}\mid x_t)$ on its own is not the simple Gaussian you started with. Bayes' rule combines the forward step with the marginal $q(x_{t-1}\mid x_0)$ , and the product of those two Gaussians is another Gaussian. That product is why a reverse step has a usable mean at all. The step being small is what keeps the product close to the true reverse conditional; with large steps the product is still Gaussian but it is no longer a good description of the reverse step.

q(x_{t-1}\mid x_t,x_0)\propto q(x_t\mid x_{t-1})q(x_{t-1}\mid x_0), \qquad \mu = \frac{\sqrt{\bar\alpha_{t-1}}\beta_t}{1-\bar\alpha_t}x_0+\frac{\sqrt{\alpha_t}(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}x_t.

The expression above still contains $x_0$ , which is exactly what you do not have at sampling time. The cumulative forward relation $x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\varepsilon$ rearranges to $x_0=(x_t-\sqrt{1-\bar\alpha_t}\varepsilon)/\sqrt{\bar\alpha_t}$ . Put that into the mean and collect the terms multiplying $x_t$ and $\varepsilon$ . The $x_t$ coefficient collapses to $1/\sqrt{\alpha_t}$ , and the noise coefficient collapses to $-\beta_t/(\sqrt{\alpha_t}\sqrt{1-\bar\alpha_t})$ . That is the whole conversion: the predicted noise is scaled by $\beta_t/\sqrt{1-\bar\alpha_t}$ before it is subtracted, and the result is divided by $\sqrt{\alpha_t}$ .

A number in the middle of the walk

Take $t=500$ on the linear schedule $\beta_t=0.0001+0.0199t/1000$ . Then $\beta_{500}=0.0001+0.0199(0.5)=0.01005$ , so $\alpha_{500}=0.98995$ and $\sqrt{\alpha_{500}}=0.99496$ . The product of the first 500 alphas is approximately $\exp(-(0.0001\cdot500+0.00000995\cdot500^2))=\exp(-2.5375)=0.0791$ , so $\sqrt{1-\bar\alpha_{500}}=\sqrt{0.9209}=0.9596$ . The noise coefficient is therefore $0.01005/0.9596=0.01047$ . If $x_t=0.5$ and the network predicts $\varepsilon=1$ , subtract $0.01047$ to get $0.48953$ , then divide by $0.99496$ to get $0.4920$ . If the prediction were $-1$ , the same arithmetic gives $0.5131$ . The mean barely moves, and the coefficient doing the moving is roughly one hundredth, not one.

The natural guess that fails

The natural guess is to subtract the predicted noise from $x_t$ and stop. That fails for two separate reasons. First, $\varepsilon_\theta(x_t,t)$ is the network's estimate of the total noise sitting in $x_t$ , not the increment of noise that the single step from $x_{t-1}$ to $x_t$ added. The factor $\beta_t/\sqrt{1-\bar\alpha_t}$ converts the total into the small increment relevant to one reverse step. Second, the forward step multiplied the signal by $\sqrt{\alpha_t}$ , so the reverse mean has to divide by $\sqrt{\alpha_t}$ to restore the scale. Skip that division and every step leaves the image slightly shrunk. Conditioning on $x_t$ alone does not give a Gaussian either; the Gaussian comes from conditioning on $x_t$ and $x_0$ and then replacing $x_0$ with the estimate above.

The formula is a Gaussian mean only under the small step assumption. Make the steps large and $q(x_{t-1}\mid x_t)$ is broad and lumpy, so a single mean and variance no longer describe it. The last step is also special: there is no later step to correct added grain, so $\sigma_t z$ is omitted at $t=1$ . The size of $\sigma_t$ is not pinned down by the forward process. Two standard choices differ by a few percent, and that difference is invisible in a thousand step sampler but becomes visible when the number of steps is cut.

Worth keeping

The reverse mean is the current image minus a scheduled fraction of the predicted noise, all divided by the square root of that step's alpha.
The predicted noise is the total noise in the current image, so it must be scaled by the step's noise coefficient before it can correct one reverse step.
The reverse step is Gaussian only because the forward steps are small; large steps destroy the single mean and variance description.

Where this goes next

This question is one step of How Noise Becomes a Picture, a course that works through its subject from the beginning. The page that teaches this part of it is Walking the Path Backwards, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course