One Filter Is Never Enough
Last timeEdges, and Skipping Positions
A filter reads the whole depth of its input and returns one number, so a layer with a hundred filters produces a hundred channels for the next layer to read.
The picture arrives with three numbers at every position, one for each colour. A
three by three filter that reads only one of them is reading a third of the
input, which is not what anybody wants. So a filter is not nine weights. It is
nine weights per input channel, stacked into a block three wide, three tall and
as deep as the input, and it reads the whole block at once.
- which filter, and therefore which output channel
- which input channel, summed over all of them
- the offsets within the window, as before
- one bias per filter, added after the sum
So one filter turns a stack of any depth into a single flat map. Depth is
consumed. If a layer is to hand the next layer something with depth of its own,
it must have several filters, and that is the whole of the arrangement: a layer
with a hundred and twenty eight filters produces a hundred and twenty eight
maps, stacked, and the next layer sees an input of depth a hundred and twenty
eight.
This makes the depth of every internal layer a free choice. The three at the
input is a fact about colour. Everything after it is a number somebody picked,
and picking it is one of the two or three decisions that actually determine
whether a design is affordable.
for o in range(Cout): # one independent filter per output channel
for r in range(Hout):
for c in range(Wout):
total = b[o]
for i in range(Cin): # the new loop: read every input channel
for u in range(k):
for v in range(k):
total += w[o][i][u][v] * x[i][r + u][c + v]
y[o][r][c] = totalWhat a layer costs
Multiply the loop lengths together and the arithmetic falls out. The output has
height times width times the number of filters positions, and each one is a dot
product over the window area times the input depth.
The lesson stops here
3 more paragraphs to go
You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.
The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.
See the planThe contentsThis is the reading half
Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.
The contents