Introduction
A convolutional neural network (CNN) is the network that revived neural networks and made deep learning work for images. This topic follows one line of thought: the traditional multilayer perceptron (MLP), the fully connected network of layers , fits its training data but fails on new data, human knowledge can restrict what the network learns, and a CNN is exactly such a restricted MLP.
Main idea
An MLP can fit any training set, but it generalizes poorly. A CNN is an MLP restricted by human knowledge: each output sees only a small window, and the same small filter is reused at every position. Every learnable weight is then trained by every pixel of every image, so it learns a general rule. Many layers widen what each output sees, and ReLU keeps the deep network trainable.
Overview
The four questions this topic answers, in order:
- how did neural networks develop, and why did they fall out of favour once already
- what is the problem of the traditional neural network, such as the MLP
- what do deep learning and a convolutional neural network compute
- how and why does a CNN solve the problem of the traditional network
Step 1: machine learning takes over more modules

An image recognition system has three modules: preprocessing and normalization, feature extraction and dimensionality reduction, then classification and decision. Two kinds of knowledge feed it: human knowledge and training samples. Human knowledge feeds all three modules, while training samples feed only the two modules that are learned. The box marked "Network" is a deep network that now covers all three modules at once.
How the role of machine learning grew:
- first, machine learning only did the classification module
- then it also did feature extraction and dimensionality reduction, such as PCA and LDA
- with deep networks (DNN, a deep neural network with many layers, and CNN) it now does all the modules at once, as one network
This raises the question that runs through the topic: is human knowledge now useless? The answer turns out to be no, because the structure of a CNN is human knowledge built into the network.
Step 2: history of neural networks

The history follows a hype cycle (expectation against time):
- single-layer networks (1950s to 1970s), then the MLP (1980s), raised expectations to a peak around 1990
- in the 2000s neural networks fell into a trough of disillusionment, for the reasons in the next note: local minima and over-fitting
- deep networks (DNN) came in 2006 and reached industry in 2009, and DNNs and CNNs are the third generation of neural networks
- instead of levelling off on the usual plateau of productivity (crossed out), the third generation keeps climbing (orange arrow)
Reading order
- Generalization Problem of Neural Networks: why does an MLP that fits its training data still fail on new data?
- Regularization with Human Knowledge: how can the learned function be restricted so that it generalizes?
- Convolutional Layer: what are the parts of a CNN, and what does one convolutional layer compute?
- CNN as a Regularized MLP: why does weight sharing with small filters, stacked deep, generalize so well?
- Further Development of CNN: how is the CNN extended to label every pixel of a scene?
Concept map
flowchart TD A["Generalization problem<br/>fitting the samples is not fitting f"] --> B["Regularization<br/>restrict f̂ with human knowledge"] B --> C["Convolutional layer<br/>small shared filters, ReLU, pooling, deep"] C --> D["CNN as a regularized MLP<br/>one MLP layer with a shared band matrix"] D --> E["Further development<br/>scene segmentation"]
Why do we need this
CNNs are the standard tool for image recognition. Understanding why they work, and not only how they compute, links them back to over-fitting in MLPs and forward to the Transformer.
Key math ideas
The math ideas this topic relies on, and where each one appears:
- Dot product and projection: a filter is dotted with the window at every position, so it is template matching across the image, in Convolutional Layer
- Gradient: a shared weight collects a gradient from every position, and the activation slope passes it back through the layers, in CNN as a Regularized MLP
- Rank: weight sharing gives each weight many more examples, the network answer to few samples in many dimensions, in CNN as a Regularized MLP
Visual 1: same layer as an MLP and as a CNN

One layer that maps a input to , trained on images:
- left: as an MLP layer it needs , about weights (biases not counted), while as a CNN layer with filters it needs , too small to see on the same axis
- right: each MLP weight meets one pixel position per image, so it gets examples, while each filter weight is used at all positions, so it gets , about
- fewer weights is the smaller family of regularization, and more examples per weight is why the CNN generalizes where the MLP over-fits
- the reading order follows this chain: the generalization problem, regularization, the convolutional layer, then the CNN as a regularized MLP