Basis Lab

Answers · How models work

What do you give up when you pool or downsample a feature map?

In shortYou give up exact position: each block of the map becomes one number, so later layers see a coarser grid, gain reach, and lose the ability to say where a response came from.

Exact position. Each small block of the map is replaced by a single number, so everything downstream sees a coarser grid whose cells each stand for several input locations, and the reach of later units grows as their ability to say where shrinks.

yr,c=max⁡0≤u,v<2x2r+u, 2c+vy_{r,c} = \max_{0 \le u,v < 2} x_{2r+u,\, 2c+v}

On an 8 by 8 map that formula produces a 4 by 4 map. Output cell (0,0)(0,0) is the largest of the four inputs (0,0)(0,0), (0,1)(0,1), (1,0)(1,0), (1,1)(1,1); whichever of them it was is not recorded anywhere, and nothing downstream can recover the difference. The value itself is exact, since the maximum of four numbers is one of those numbers, so what is lost is not precision but the coordinate. Halve again and each cell stands for sixteen inputs. Start from a 32 pixel side and the side lengths run 32, 16, 8, 4, 2, 1, so after five halvings a single output position covers 32×32=102432 \times 32 = 1024 input positions. Four positions become one, then sixteen, then sixty four, and the map has no way of marking which of them fired.

Reach is the other side of the trade

The same halving that merges cells widens the region a later unit sees, without a single extra weight. A 3 by 3 window at stride one on the reduced map touches 3 positions. At the original resolution those 3 positions would have been 3 pixels. After one halving each position stands for 2 pixels, so the window covers 6. After a second halving each position stands for 4, so it covers 12. Reach doubles at every halving while the number of positions quarters, which is why the arithmetic in each stage stays roughly flat as the map shrinks: the cost of a 3 by 3 convolution on a map of side ss is 9s29s^2 multiplies per channel pair, and at side s/2s/2 it is 9(s/2)29(s/2)^2, exactly a quarter.

FIG 1
halvingsside lengthinput positions per cellinput pixels spanned by a 3 by 3 window
03213
11646
281612
346424
4225648

The natural guess is that this is purely a saving: the same information, less arithmetic. The arithmetic saved is real, a quarter of the multiplies in every layer that follows, but the tolerance to small shifts and the growth in reach are not bought with that saving. They are bought with the discarding. Four values enter each cell and one leaves, and the identity of the winner leaves with it. The backward pass makes the same point: the whole gradient goes to whichever input was largest and the other three get nothing, which is a routing decision that records exactly one position as having mattered and treats the rest as indistinguishable from it. The information was not compressed, it was dropped.

The shift tolerance that this buys is real but approximate, which is the part usually overstated. Lay a row of values 5, 0 over the block {0,1}\{0,1\}: the max is 5. Move the pattern one place right to 0, 5: still in the same block, the max is still 5, the output has not moved at all. Move it once more, to 0, 0, 5: the 5 now sits in block {2,3}\{2,3\}, whose max is 5, while the first block's max has fallen to 0, and the output has changed completely. Whether a one pixel shift is absorbed depends on where the pattern started relative to a boundary fixed by the sampling grid, not by the content of the picture. That is why a one pixel shift of an input can change a trained classifier's answer, and why blurring before subsampling is the standard repair: the blur removes the hard boundary that the sampling would otherwise expose.

Where the trade stops paying

For deciding whether a photograph contains a dog, the lost coordinate is not merely tolerable but helpful, because the answer does not depend on it and the coarser map makes the decision easier. For outlining the dog, counting overlapping dogs, or saying which pixels belong to one, an output cell standing for 32 pixels on each side is far past the resolution the answer needs. The map has a receptive field wide enough to know there is a dog and no coordinate precise enough to say where its edge lies.

Any task that needs the discarded position has to put it back, and the standard approaches differ in what they do with the coordinate. One keeps the high resolution maps on a separate path and merges them into the coarse, deep ones, which is the shape of almost every segmentation network. One refuses to reduce as much and widens the windows instead, spreading a 3 by 3 to cover more input without dropping any position. One reduces and then undoes it, learning an upsampling that uses the recorded positions of the maxima to decide where each coarse value belongs, which is an admission that the coordinate had to be kept after all. Most recent designs also replace the fixed maximum with a convolution at stride two: the map is still halved, but with weights that are learned, so how the reduction is done is trained rather than chosen, and the hard sampling is replaced by something closer to a blur along the way down.

Worth keeping

  • A 2 by 2 pooling halves each side and quarters the area, so after five halvings one output position stands for 1024 input positions.
  • Reach doubles at every halving while the number of positions quarters, which keeps the arithmetic per stage roughly constant as the map shrinks.
  • Shift tolerance from pooling is approximate: a one pixel shift is absorbed only when the pattern stays inside its block, and the block boundaries come from the sampling grid rather than the content.

Where this goes next

This question is one step of How a Filter Reads a Picture, a course that works through its subject from the beginning. The page that teaches this part of it is Discarding Positions on Purpose, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. What does it mean to use stride 2 in a convolution?Using stride 2 keeps the filter the same but moves its window two positions at a time, so it produces about half as many outputs in each direction and about a quarter of the work in two dimensions.
  2. Why do CNNs use the same weights everywhere instead of learning a separate filter for each spot?They share one small filter across every position because a pattern such as an edge means the same thing wherever it appears; the huge cut in parameters is a consequence, not the main reason.
  3. Why does a convolution make the image smaller if you don't pad it?A convolution shrinks the image because the window must fit completely inside it, so the first and last few positions have no full window. Padding adds extra values at the border so those positions can be used.
  4. What does a convolution actually calculate at a single point in the image?It multiplies each filter weight by the pixel beneath it, adds those products, and writes the single total at the matching position in the output.
  5. Why does the model get stuck repeating the same sentence over and over?The model is not stuck in a memory loop: once it has written a phrase, that phrase is in its context and makes writing it again more likely, so a decoder that always picks the top word cannot leave.
  6. Why does diffusion add Gaussian noise instead of any random corruption?Diffusion adds Gaussian noise because Gaussian errors combine exactly: each step keeps the spread fixed and gives a closed-form noisy picture at any later step, which makes the reverse step learnable.

Read alongside