Where the Window Fits, and How Often It Stops
Last timeWhy the Same Weights Everywhere
Two small choices decide the size of what comes out: what to do where the window hangs off the edge, and whether to use every position or every second one.
A three-wide window on a row of a hundred values fits in ninety eight places,
not a hundred. Two positions have been lost, one at each end, because at the very
first position the window would need a value to the left of the first and there
is none. In one dimension with a three-wide window that is a trivial loss. Stack
twenty layers of it in two dimensions and a two hundred and twenty four pixel
picture has shrunk to a hundred and eighty four, for no reason anybody intended.
- the input size along one axis, in positions
- the padding added to each side, so 2p in total
- the window width along that axis
- the stride, how far the window moves between uses
- round down, because a partial step does not give a position
The rounding down is where the surprises live. If the stride does not divide the
available span, the last partial step is simply not taken, and a strip at the far
edge of the input is read by no window at all. Nothing warns about this. It is
the reason two implementations of the same architecture can disagree by one pixel
and the reason a network sometimes behaves slightly differently on the right
side of an image than the left.
| input size | window | padding | stride | output size | |
|---|---|---|---|---|---|
| no padding, every positi | 32 | 3 | 0 | 1 | 30 |
| size preserving | 32 | 3 | 1 | 1 | 32 |
| size preserving, every s | 32 | 3 | 1 | 2 | 16 |
| wider window, every seco | 32 | 5 | 2 | 2 | 16 |
| no padding, every third | 32 | 3 | 0 | 3 | 10 |
Padding, and the two reasons for it
The first reason is the one just given. Without padding, every layer shrinks the
picture, and in a deep network that shrinkage compounds until there is nothing
left to convolve. Padding of one with a three-wide window gives back exactly the
two positions that were lost.
The lesson stops here
4 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents