Answers · How models work

What does classifier-free guidance actually add together, and why does it make the prompt stronger?

In shortIt adds the unconditional prediction to several times the gap between the conditional and unconditional predictions, pushing past the conditional answer so the prompt is obeyed more strictly.

Classifier-free guidance adds the unconditional prediction to $w$ times the difference between the conditional and the unconditional prediction, and hands that sum to the sampler in place of either. The prompt becomes stronger because the difference is the direction the condition pushed the prediction in, and the sampler is made to walk further along that direction than the conditional prediction alone went.

\tilde{\varepsilon} = \varepsilon_\theta(x_t, t, \varnothing) + w\left(\varepsilon_\theta(x_t, t, c) - \varepsilon_\theta(x_t, t, \varnothing)\right)

Read the two endpoints. At $w=0$ the bracket is multiplied away and what remains is exactly the unconditional prediction, so the condition has no effect at all. At $w=1$ the unconditional term cancels against itself and what remains is exactly the conditional prediction, which is the model sampled as it was trained. Every value between zero and one is a straight interpolation between ignoring the condition and honouring it. Every value above one is an extrapolation past anything training produced.

\tilde{\varepsilon} = 0.30 + 7.5\,(0.50 - 0.30) = 0.30 + 7.5 \times 0.20 = 0.30 + 1.50 = 1.80

Take one component of the noise prediction at one step. Asked with the condition, the network says 0.50. Asked with the blank, it says 0.30. The difference, 0.20, is what the condition contributed to that component. At the usual default the number passed to the sampling step is 1.80, which is 1.30 beyond the conditional answer. The same arithmetic at $w=1$ returns 0.50 and at $w=0$ returns 0.30, so the two anchors are intact. Nothing downstream knows that 1.80 is an extrapolation: it is treated as an ordinary noise prediction and the step is taken from it.

Why not just scale the conditional prediction

A bigger prompt weight suggests multiplying the conditional prediction by some factor, $w\,\varepsilon_\theta(x_t,t,c)$ . That fails for a reason the subtraction exposes. Many components of the prediction are untouched by the condition: ask the network with and without it and the answers are identical there. The difference is then zero, and the guided prediction is that common value for every $w$ , however large. Multiplying would inflate those components anyway, even though the condition said nothing about them. Taking the difference and adding it back amplifies only what changed when the condition was supplied, which is why the strength can be pushed high without the whole prediction blowing up uniformly.

What the overshoot does to the distribution

The extrapolation has an exact reading in terms of distributions. The point produced by that formula is a sample from $\tilde{p}(x \mid c) \propto p(x)\,p(c \mid x)^{w}$ , where $p(c \mid x)$ is how well a picture agrees with the condition as the model implies it. The strength has moved out of the multiplier and into the exponent.

\tilde{p}(x \mid c) \;\propto\; p(x)\, p(c \mid x)^{w}

FIG 1

strength			ratio
0	1.000	1.000	1.0
1	0.900	0.600	1.5
5	0.590	0.078	7.6
7.5	0.454	0.022	21
20	0.122	0.000037	3300

At strength one the picture that agrees better is one and a half times as likely as the other. At 7.5 it is about 21 times as likely, and at 20 about 3300 times. Raising a probability to a power above one always does this: it favours whatever already scored highest and pushes everything else down. So guidance does not teach the model anything new about the condition. It restricts sampling to the least ambiguous members of the conditioned set, and the pictures that match only loosely, the dog half out of frame, stop being drawn almost entirely. That is what reads as a stronger prompt, and the loss of variety is the same operation seen from the other side, not a side effect of it.

There is a computational price as well. Every sampling step runs the network twice, once conditioned and once with the blank, so a 50 step sampler makes 100 forward passes instead of 50. The two calls can be batched into one pass through the hardware, which improves the wall clock but not the arithmetic.

Where it stops being true

Past a moderate strength the difference between the two predictions stops being a reliable direction, because the extrapolated point has left the region where the linear step meant anything. The pictures then acquire a recognisable look: excessive contrast, oversaturated colour, an oddly literal reading of the words. The usual patch is to rescale the guided prediction back towards the magnitude of the conditional one, which works and is an admission that the step went too far. None of this touches the $w=1$ case, where nothing is extrapolated and nothing needs rescaling.

Worth keeping

Multiplying the conditional prediction by a constant would inflate parts of the prediction the condition never influenced; the difference between the conditional and unconditional predictions isolates exactly what the condition changed.
The guided prediction equals the conditioned model's own output only at a strength of one, so every setting above one is a deliberate distortion of the distribution the model learned.
Guidance sharpens rather than improves: it suppresses the loosely matching pictures by raising agreement with the condition to a power.

Where this goes next

This question is one step of How Noise Becomes a Picture, a course that works through its subject from the beginning. The page that teaches this part of it is Steering the Walk, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course