Why share weights?
A dense layer learns a different weight for every input-output connection. An image edge should be recognisable in many positions, so a convolutional layer reuses one small kernel over the spatial grid. Fukushima’s neocognitron (1980) combined local receptive fields and hierarchical feature extraction; LeCun and collaborators trained convolutional networks with backpropagation for handwritten ZIP codes in 1989.
A 3 × 3 kernel takes a weighted sum of nine pixels. CNN libraries usually call this convolution even though they do not flip the kernel; mathematically it is cross-correlation. Each filter makes one feature map. Multiple input channels contribute sums to that map, with a learned bias added once.
Stride, padding and shape
With W = 8, K = 3, stride S = 1 and no padding, the output is 6 × 6. Padding P = 1 preserves 8 × 8. Stride 2 skips positions, reducing resolution. Outside-image entries are zero; they participate in the displayed products but do not correspond to editable pixels.
Enable JavaScript to use this local playground. The explanation and sources below work without it.
Receptive fields and pooling
Choose Sobel horizontal, then invert a pixel on a horizontal edge. The output responds to brightness changes from top to bottom; Sobel vertical responds to left-right changes. A blur averages nearby pixels, while sharpen amplifies their difference from the centre.
Stacking stride-1 3 × 3 convolutions increases the receptive field: two layers can see a 5 × 5 input patch. More generally, if a layer has receptive field r and jump j between centres, the next kernel grows r by (K − 1)j and multiplies j by its stride. Padding changes border behaviour, not the theoretical receptive-field size.
Max pooling selects the largest value in a window; it adds no learned weights. Open the pooling panel to inspect 2 × 2 windows of the complete output. Pooling discards location and sign information; a full CNN often applies ReLU before pooling, but this demonstration pools the raw signed convolution so the operations remain distinguishable.
Convolutional Neural Networks: Spatial Priors, Weight Sharing & Pooling
Why not feed images directly into a dense MLP? Consider a modest 256 × 256 RGB image (196,608 inputs). A single dense layer with 1,000 hidden units requires 196.6 million weights—and worse, if an object shifts by just 2 pixels, every input lands on a completely different weight, forcing the network to re-learn edge detectors at every (x, y) coordinate from scratch.
Convolutional Neural Networks—pioneered by Fukushima's Neocognitron (1980), LeCun's LeNet-5 (1989/1998) for handwritten zip codes, and scaled to GPUs by Krizhevsky, Sutskever, and Hinton's AlexNet (2012)—encode two physical priors of natural signals directly into the architecture:
-
Locality & Sparse Connectivity: Early visual features (oriented edges, corners, strokes) depend only on a small local neighborhood
k × k(here3 × 3pixels). Stacking convolutional layers grows the effective receptive field linearly with depth (two3 × 3layers see a5 × 5window; three see7 × 7). -
Spatial Weight Sharing (Translation Equivariance): The exact same
3 × 3kernel tensorK(f)(only 9 weights + 1 bias!) is slid across every rowrand columncof the image. Shifting the input shifts the resulting feature map identically:f(TΔr,Δc I) = TΔr,Δc f(I). -
Local Max-Pooling & Switch Routing: A
2 × 2stride-2 MaxPool layer outputs the maximum activation in each2 × 2patch, granting local translation invariance. During backpropagation, MaxPool acts as a router: it records theargmaxindex during the forward pass and routes 100% of the upstream gradient∂L/∂pexclusively to that winning pixel while zeroing the other three.
Draw or Load an 8×8 Glyph & Train Kernels
Click cells to toggle pixels, or load a preset pattern to inspect how each 3×3 filter responds.