The convolutional neural network (ConvNet or CNN) is an evolution of the multilayer perceptron that replaces matrix multiplication with convolution in at least one layer^{[GBC16, Ch. 9]}.

A deep CNN is a CNN that has more than three hidden layers.

CNNs play an important role in the history of deep learning because^{[GBC16, §9.11]}:

• They exemplify successful applications of neuroscientific insights^[Lin21] to machine learning.
• CNNs are among the first neural networks to achieve commercial success; for example, AT&T used the LeNet-5 CNN to read bank checks in the 1990s^[LBBH98].
• Deep CNNs are among the first to be successfully trained with backpropagation.
• Deep CNNs are among the first deep neural networks to achieve state-of-the-art results for challenging problems; for example, the ground-breaking performance of the 8-layer CNN called AlexNet in the 2012 ImageNet challenge^[KSH17] is widely considered to be the watershed event that propelled deep learning research.
• Deep CNNs remain pertinent for contemporary applications; for example, a combination of CNN and LSTM has been applied to predicting aircraft trajectory for assisting air crews in their decision-making during the approach phase of a flight^[LPM21].

The following discusses the basic CNN structure and zooms in on the structural elements.

Structure

Invariance to local translation is useful if detecting the presence of a feature is more important than localizing the feature.

When it was introduced, the CNN brought three architectural innovations to achieve shift/translation-invariance^{[LB02, GBC16]}:

Structural element: convolution

It has to be emphasised that the convolution in this context is inspired by but not equivalent to the convolution in linear system theory and furthermore it exists in several variants^{[GBC16, Ch. 9]}, but it is still a linear operation.

In signal processing, if $x(t)$ denotes a time-dependent input and $f(t)$ denotes the impulse response function of a linear system, then the output response of the system is given by the convolution (denoted by symbol $\ast$) of $x(t)$ and $f(t)$:

$(x\ast f)(t) = \int_{\tau=0}^t x(\tau)f(t-\tau)d\tau.$

In discrete time, the equation above can be written as

$(x\ast f)[t] = \sum_{\tau=0}^t x[t]\cdot f[t-\tau].$

Above, square brackets are used to distinguish discrete time from continuous time.

In machine learning (ML), $f(t)$ is called a kernel or filter and the output of convolution is an example of a feature map.

For two-dimensional (2D) inputs (e.g., images), we use 2D kernels in convolution:

Convolution works similarly to windowed (Gabor) Fourier transforms^{[BK19, §6.5]} and wavelet transforms^[Mal16].

Convolution is not to be confused with cross-correlation, but for efficiency, convolution is often implemented as cross-correlation in ML libraries. For example, PyTorch implements 2D convolution (Conv2d) as cross-correlation (denoted by symbol $\star$):

Note above, the indices of $f$ are $a$ and $b$ instead of $i-a$ and $j-b$. The absence of flipping of the indices makes cross-correlation more efficient.

Structural element: pooling

A pooling function replaces the output of a neural network at a certain location with a summary statistic (e.g., maximum, average) of the nearby outputs^{[GBC16, §9.3]}. The number of these nearby outputs is the pool width. Like a convolution/cross-correlation filter, a pooling window is slid over the input one stride at a time.

Fig. 5 illustrates the maximum-pooling (max-pooling for short) operation^[RP99].

Pooling helps make a feature map approximately invariant to small translations of the input. Furthermore, for many tasks, pooling is essential for handling inputs of varying size.

Fig. 6 illustrates the role of max-pooling in downsampling.

As a summary, Fig. 7 shows the structure of an example of an early CNN^[LB02]. Note though the last layers of a modern CNN are typically fully connected layers (also known as dense layers).