ContentsThe library

How Noise Becomes a Picture

Overshooting the Condition on Purpose

Last timeWhat the Prediction Really Is

A condition handed to the network is honoured too weakly, and the standard fix is to measure its effect on the prediction and then deliberately overshoot it.

Everything so far generates pictures with no say in what they contain. Adding a

condition, a class or a line of text, is easy enough: pass it to the network as

an extra input and train as before. What is surprising is that this works

correctly and is not good enough.

The condition is honoured on average

A model trained this way learns the distribution of pictures that match the

condition, and in any real collection that distribution is loose. Images

captioned as a dog include photographs where a dog is one object among many,

poorly lit, or half out of frame. Sampling faithfully reproduces all of it. The

condition is satisfied in the aggregate and frequently not in the sample on the

screen.

FIG 1One network, trained for both jobs
The trick is that the unconditional model is free. It was already being trained, on a tenth of the data, by a procedure that looks like nothing more than a dropout regulariser.
FIG 2The guided prediction
the prediction with the condition supplied
the prediction with the condition replaced by a blank
the strength, typically between 5 and 10 for text conditioned models
what is handed to the sampling step in place of a single prediction
At a strength of zero this is the unconditional prediction and the condition is ignored. At one it is exactly the conditional prediction, which is the model sampled faithfully. Above one it is a point beyond the conditional prediction, further along the direction in which supplying the condition moved the answer. No training produced that point; it is an extrapolation.
FIG 3What the strength dial does
interpolating, and nobody works hereextrapolating, and everybody works here510150condition ignored1the model, sampled faithfully7.5the usual default20visibly overdoneguidance strength
The two spans are the uncomfortable fact about this method. The only setting that samples the trained model honestly is one, and every system in production runs at five or above. What is shipped as generation is therefore a deliberate distortion of the distribution the model learned, tuned by eye.
FIG 4The distribution actually sampled
how plausible the picture is on its own terms, with no reference to the condition
how well the picture agrees with the condition, as implied by the model
the strength, now visible as an exponent rather than a multiplier
This is the real content of the method. Raising a probability to a power above one is a sharpening operation: whatever already scored highest is favoured further and everything else is pushed down. So guidance does not make the model better at honouring conditions, it restricts sampling to the least ambiguous members of the conditioned set, and anything that only loosely matches stops appearing.
FIG 5
StrengthAgreement with the conditionVariety of outputOverdone appearanceTwo calls per step
0ignoredfullnoneno
1loosefullnoneno
7.5tightreducedslightyes
20very tightpoorstrongyes

What the strength is doing to the distribution

The honest setting is the second row, and its loose agreement with the condition is

what people complain about. The last row is what happens when the complaint is

answered too enthusiastically. The usual default sits in the third row and is a

judgement about taste, not a value anything derives.

The lesson stops here

2 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the planThe contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

The rest of this course

  1. 01A Thousand Small Steps From a Photograph to Static
  2. 02A Thousand Steps Collapse Into One Lineopening only
  3. 03Five Lines of Training, and One Squared Erroropening only
  4. 04One Step Back, a Thousand Timesopening only
  5. 05The Network Is Pointing Uphillopening only
  6. 06Overshooting the Condition on Purposeyou are here
  7. 07The Same Model, Twenty Times Fasteropening only
  8. 08If the Path Is Yours to Pick, Pick a Lineopening only

Read alongside