Answers · How models work

What does a convolution actually calculate at a single point in the image?

In shortIt multiplies each filter weight by the pixel beneath it, adds those products, and writes the single total at the matching position in the output.

At one position, a convolution multiplies each weight in the filter by the input value under it, adds those products, and writes that one number into the output at the same position. It is a dot product between the filter and one local patch, so the number measures how much that patch agrees with the filter's pattern. The result is one value, not a patch and not a new picture of the whole input.

y_i = \sum_{j=0}^{k-1} w_j\, x_{i+j}

Take the filter $w=(-1,0,+1)$ and the row $x=(10,10,90,90,20)$ . The first window starts at $i=0$ and reads $(10,10,90)$ : the product is $(-1)(10)+(0)(10)+(1)(90)=-10+0+90=80$ . The window starting at $i=1$ reads $(10,90,90)$ : $(-1)(10)+(0)(90)+(1)(90)=-10+0+90=80$ . The window starting at $i=2$ reads $(90,90,20)$ : $(-1)(90)+(0)(90)+(1)(20)=-90+0+20=-70$ . Each window used the same three weights; only the three values underneath changed.

FIG 1

start	patch	products	response
0	10, 10, 90	-10, 0, 90	80
1	10, 90, 90	-10, 0, 90	80
2	90, 90, 20	-90, 0, 20	-70

The blur guess

The natural guess is that the output is a blurred copy of the input, as if every filter averaged nearby pixels. A blur is a dot product too, but with weights that are all positive and sum to one: with weights $(1/4,1/2,1/4)$ on a flat patch $(10,10,10)$ the output is $(1/4)10+(1/2)10+(1/4)10=2.5+5+2.5=10$ . Now use $(-1,0,+1)$ on the same flat patch: $-10+0+10=0$ . The weights sum to $-1+0+1=0$ , so flat regions vanish no matter how bright they are. The output cannot be a copy of brightness. It is the amount of agreement between the patch and the filter, which is why a change from dark to bright gives $80$ while a change in the other direction gives $-70$ .

y_{r,c} = \sum_{u=0}^{k_h-1} \sum_{v=0}^{k_w-1} w_{u,v}\, x_{r+u,\,c+v}

A two-dimensional window changes the number of sums, not the idea. A $3$ by $3$ filter has $3\times 3=9$ weights, lies over $9$ input values at a time, and produces one number from those $9$ products. The row index and the column index of the output say where the patch was, because the value stored there came from the input centred near that same row and column.

The output is a map

On the row above, the output is $(80,80,-70)$ , one value for each valid window start. The first two values mark the places where the row rises from $10$ towards $90$ ; the third marks the place where it falls from $90$ to $20$ . That ordering is the point: the output has the same geography as the input. A layer that turned the whole row into a few numbers would have thrown the geography away, but a convolution writes each response at the place that caused it, so the result is a map of where the pattern occurred.

A dot product is large when two lists are large and point the same way, and near zero when they disagree. The filter is a list of weights, so it is a direction in the space of small patches. The response at a position asks how much of the patch beneath the window lies along that direction. That is why the same three weights can find an edge without knowing the word edge: the weights are the pattern, and the dot product measures resemblance to it.

The operation is also linear: double the row to $(20,20,180,180,40)$ and the first window gives $(-1)(20)+(0)(20)+(1)(180)=-20+0+180=160$ , which is twice $80$ . Shift the input and the responses shift with it, unchanged in value. This is the precise sense in which all positions are treated alike, and it is what a connected layer lacks, since there every position has its own weights.

Where the reading changes

The formula assumes the patch exists. At an edge the indices run past the input, so something has to be chosen: pad the input, or drop the positions that do not fit. Stride changes the step between window starts, and dilation spaces out which input values the weights read, but neither changes the dot product itself. In a deeper layer the patch is no longer made of pixels but of earlier feature values, so the filter is matching a pattern in a feature map rather than a pattern in the picture. And if the weights are allowed to differ from position to position, the operation stops being a convolution and becomes a connected layer. The blur reading is only the special case where every weight is positive and the weights sum to one.

Worth keeping

At one position a convolution is a dot product between the filter and the patch under it.
The same weights slide to every position, so the output is a map of where the pattern matched, not a summary of the whole input.
A blur is the special case where all weights are positive and sum to one; an edge filter has weights summing to zero and gives zero on flat patches.

Where this goes next

This question is one step of How a Filter Reads a Picture, a course that works through its subject from the beginning. The page that teaches this part of it is One Small Window, Moved Everywhere, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course