Convolutional Networks for Images

Take a 224 × 224 color photo and feed it into a dense layer of 1,000 units. Do the math: $224 \times 224 \times 3 = 150{,}528$ inputs, multiplied by 1,000 outputs, plus the biases. A hundred and fifty million parameters. For a single layer, on a single image of modest resolution. Six hundred megabytes of weights before the model has learned anything at all.

Yet the problem isn't primarily one of size. It's that this model treats the image as a bag of 150,528 independent numbers. The pixel in the top-left corner and its immediate neighbor are, to the model, as unrelated as two pixels in opposite corners. All the structure of the image, the fact that neighborhoods matter and that an edge is an edge wherever it appears, is thrown away before the first computation. The network will have to relearn it from examples, which is possible but requires an absurd amount of data to rediscover something we already knew.

Convolutional networks build this knowledge into the architecture. They rely on just two assumptions: useful patterns are local, and a pattern that is useful in one place is useful elsewhere too. The first replaces full connectivity with a small neighborhood. The second lets the same weights be reused everywhere in the image. Together, these two ideas bring 150 million parameters down to a few thousand, and above all they give the model a structure it no longer has to guess.