Answers · How models work

Why do CNNs use the same weights everywhere instead of learning a separate filter for each spot?

In shortThey share one small filter across every position because a pattern such as an edge means the same thing wherever it appears; the huge cut in parameters is a consequence, not the main reason.

Convolutional networks reuse one small filter at every location because the rule that turns a patch into a feature should not depend on where the patch is. The parameter saving is real, but it is the arithmetic consequence of that constraint rather than the reason for it.

256^2 = 65{,}536,\qquad 3\cdot256^2 = 196{,}608,\qquad (196{,}608)^2 = 38{,}654{,}705{,}664.

Take a colour photograph 256 pixels on a side. It has 65,536 positions and, with three colours, 196,608 input numbers. A connected layer with as many outputs as inputs needs one weight for each input and output pair, so the count is 196,608 squared, which is 38,654,705,664, about forty billion. That is not a layer one can train on any ordinary data set.

y_{r,c}=\sum_{i,j} W_{r,c,i,j}\,x_{i,j}+b_{r,c}\qquad\text{versus}\qquad y_{r,c}=\sum_{u=0}^{2}\sum_{v=0}^{2} w_{u,v}\,x_{r+u,c+v}+b.

In the first expression $W$ carries $r$ and $c$ , the output position, so every location has its own weights. In the second $w$ carries only $u$ and $v$ , offsets inside a three by three window, and the output position appears only on $x$ . The same nine numbers are used at every $r$ and $c$ . The picture size is absent from the parameter count: a three by three greyscale filter has $3\times3=9$ weights, and a colour layer with 64 filters has $3\times3\times3\times64+64=1{,}792$ weights. The image side length never enters.

It is worth separating the constraints. Restricting each output to a small window is locality. On its own it cuts the weight count a great deal but still leaves each position its own numbers. Forcing all positions to use the same numbers is sharing. A convolution applies both, and the question is about the second.

The saving is the smaller half

It is tempting to read this as a compromise: the connected layer is what one would want, sharing is what one can afford, and something is lost. That reading predicts that convolution would fade as memory became cheaper. It has not, because sharing also encodes a claim about pictures. A vertical edge is a vertical edge in the top left or the bottom right; what makes it an edge is the local arrangement of brightness, and that arrangement means the same thing in both places. A connected layer does not know this. It can learn an edge detector for the top left that fails in the top right because the training set happened not to contain an edge there. Sharing forces one detector to work everywhere, which is the correct constraint for most photographic evidence.

Every position is another example

The constraint also changes the learning problem. A weight used at one position only sees whatever appeared at that position, so 65,536 positions are 65,536 little problems, each with as many examples as there are images. A shared weight sees every window position in every image. Ten thousand images of 256 pixels a side give it $10{,}000\times65{,}536=655{,}360{,}000$ patches, about six hundred and fifty million examples for the same nine numbers. The effective number of examples per weight is larger by the number of positions, which is why a convolutional network can generalise from a training set that would be hopeless for a connected layer of similar capability.

What sharing forbids

The constraint is real and it forbids things. Nothing in a shared filter can respond differently in the middle of a picture than at the edge, because no term in the arithmetic knows where the window is. A task that genuinely depends on absolute position is therefore invisible to a stack of shared filters, however deep. Asked to paint a dot at a given pair of coordinates, such a network fails in a way that looks absurd for a network that can recognise many kinds of object. The repair is not to abandon sharing. It is to hand the network extra input channels holding the row and column index of each position. Sharing stays, and the filters can read position as ordinary input data.

When sharing is wrong for some part of a problem, the fix is to supply as data whatever the constraint refuses to represent.

Worth keeping

Sharing a filter across positions asserts that a pattern means the same thing wherever it appears, which is true of most photographic evidence.
A dense layer needs a weight for every input pixel of every output unit, so a colour 256 by 256 image with as many outputs as inputs needs about forty billion weights.
Sharing forbids absolute position, and when position matters the repair is to add position as input channels rather than to abandon sharing.

Where this goes next

This question is one step of How a Filter Reads a Picture, a course that works through its subject from the beginning. The page that teaches this part of it is Why the Same Weights Everywhere, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course