ContentsThe library

How Noise Becomes a Picture

One Step Back, a Thousand Times

Last timeThe Only Thing Being Trained

The reverse of a small forward step has the same simple shape, and the trained network supplies the one quantity needed to point it in the right direction.

The previous lesson trained a network to answer one question: given a noised

picture and a time, which noise was added. This lesson turns that answer into a

picture. Nothing further is trained, and nothing further is derived about the

network. All that is left is arithmetic.

Why the steps had to be small

The forward process was built from a thousand tiny steps rather than a handful

of large ones, and this is where that choice is repaid. A step that barely moves

the picture leaves a narrow set of pictures that could have produced the result,

and over a narrow set the reverse step has the same simple shape as the forward

step: a centre and a small spread around it. That claim is the whole foundation,

and it fails for large steps, where the set of possible origins is broad and

lumpy and no simple shape describes it.

FIG 1One reverse step
where the walk currently is, starting from pure noise
the network's single guess at the noise present in it
the fraction of that guess actually removed, which is small because the step size is small
undoing the shrink the forward step applied to the signal
a fresh draw of noise, scaled, added back at every step except the last
Read it as three separate moves: remove a sliver of what the network calls noise, restore the scale the forward step took away, then put a little noise back. The network appears once, and only in the middle term.
FIG 2How much of the prediction each step removes
0.010.010.010.020.031.0250.8500.5750.31000.0time, from the clean end to the noisy end
straight line schedule
Nowhere does this reach anything close to one, which is the point. Even at the noisy end, where the network has the clearest view of the noise because there is little else present, a single step removes only a few percent of it. The thousand steps are not redundancy; each one is deliberately almost nothing.

Why a fresh draw at every step

If each step moved only to its centre, the loop would be following a sequence of

conditional averages, and the lesson before this one explained what those are:

composites of everything still consistent. Following them without interruption

walks towards the average picture and arrives at a smooth, characterless image,

the same one every time.

The lesson stops here

3 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the planThe contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

The rest of this course

  1. 01A Thousand Small Steps From a Photograph to Static
  2. 02A Thousand Steps Collapse Into One Lineopening only
  3. 03Five Lines of Training, and One Squared Erroropening only
  4. 04One Step Back, a Thousand Timesyou are here
  5. 05The Network Is Pointing Uphillopening only
  6. 06Overshooting the Condition on Purposeopening only
  7. 07The Same Model, Twenty Times Fasteropening only
  8. 08If the Path Is Yours to Pick, Pick a Lineopening only

Read alongside