Introduction

A convolutional neural network (CNN) is the network that revived neural networks and made deep learning work for images. This topic follows one line of thought: the traditional multilayer perceptron (MLP), the fully connected network of layers , fits its training data but fails on new data, human knowledge can restrict what the network learns, and a CNN is exactly such a restricted MLP.

Main idea

An MLP can fit any training set, but it generalizes poorly. A CNN is an MLP restricted by human knowledge: each output sees only a small window, and the same small filter is reused at every position. Every learnable weight is then trained by every pixel of every image, so it learns a general rule. Many layers widen what each output sees, and ReLU keeps the deep network trainable.


Overview

The four questions this topic answers, in order:

  • how did neural networks develop, and why did they fall out of favour once already
  • what is the problem of the traditional neural network, such as the MLP
  • what do deep learning and a convolutional neural network compute
  • how and why does a CNN solve the problem of the traditional network

Step 1: machine learning takes over more modules

An image recognition system has three modules: preprocessing and normalization, feature extraction and dimensionality reduction, then classification and decision. Two kinds of knowledge feed it: human knowledge and training samples. Human knowledge feeds all three modules, while training samples feed only the two modules that are learned. The box marked "Network" is a deep network that now covers all three modules at once.

How the role of machine learning grew:

  • first, machine learning only did the classification module
  • then it also did feature extraction and dimensionality reduction, such as PCA and LDA
  • with deep networks (DNN, a deep neural network with many layers, and CNN) it now does all the modules at once, as one network

This raises the question that runs through the topic: is human knowledge now useless? The answer turns out to be no, because the structure of a CNN is human knowledge built into the network.


Step 2: history of neural networks

The history follows a hype cycle (expectation against time):

  • single-layer networks (1950s to 1970s), then the MLP (1980s), raised expectations to a peak around 1990
  • in the 2000s neural networks fell into a trough of disillusionment, for the reasons in the next note: local minima and over-fitting
  • deep networks (DNN) came in 2006 and reached industry in 2009, and DNNs and CNNs are the third generation of neural networks
  • instead of levelling off on the usual plateau of productivity (crossed out), the third generation keeps climbing (orange arrow)

Reading order

  1. Generalization Problem of Neural Networks: why does an MLP that fits its training data still fail on new data?
  2. Regularization with Human Knowledge: how can the learned function be restricted so that it generalizes?
  3. Convolutional Layer: what are the parts of a CNN, and what does one convolutional layer compute?
  4. CNN as a Regularized MLP: why does weight sharing with small filters, stacked deep, generalize so well?
  5. Further Development of CNN: how is the CNN extended to label every pixel of a scene?

Concept map

flowchart TD
    A["Generalization problem<br/>fitting the samples is not fitting f"] --> B["Regularization<br/>restrict f̂ with human knowledge"]
    B --> C["Convolutional layer<br/>small shared filters, ReLU, pooling, deep"]
    C --> D["CNN as a regularized MLP<br/>one MLP layer with a shared band matrix"]
    D --> E["Further development<br/>scene segmentation"]

Why do we need this

CNNs are the standard tool for image recognition. Understanding why they work, and not only how they compute, links them back to over-fitting in MLPs and forward to the Transformer.


Key math ideas

The math ideas this topic relies on, and where each one appears:

  • Dot product and projection: a filter is dotted with the window at every position, so it is template matching across the image, in Convolutional Layer
  • Gradient: a shared weight collects a gradient from every position, and the activation slope passes it back through the layers, in CNN as a Regularized MLP
  • Rank: weight sharing gives each weight many more examples, the network answer to few samples in many dimensions, in CNN as a Regularized MLP

Visual 1: same layer as an MLP and as a CNN

One layer that maps a input to , trained on images:

  • left: as an MLP layer it needs , about weights (biases not counted), while as a CNN layer with filters it needs , too small to see on the same axis
  • right: each MLP weight meets one pixel position per image, so it gets examples, while each filter weight is used at all positions, so it gets , about
  • fewer weights is the smaller family of regularization, and more examples per weight is why the CNN generalizes where the MLP over-fits
  • the reading order follows this chain: the generalization problem, regularization, the convolutional layer, then the CNN as a regularized MLP