This knowledge base entry follows discussion of artificial neural networks and backpropagation.

Contemporary options for $\varphi$ are the non-saturating activation functions [Mur22, Sec. 13.4.3], although the term is not accurate.

Below, $x$ should be understood as the output of the summing junction.

• The rectified linear unit (ReLU) [NH10] is the unipolar function:

  $\varphi(x) = \text{ReLU}(x) \triangleq \max(0,x) = \begin{cases} x & \text{if }x>0, \\ 0 & \text{otherwise.} \end{cases}$

  ReLU is differentiable except at $x=0$, but by definition, $\text{ReLU}'(x)=0$ for $x\leq0$.

  ReLU has the advantage of having well-behaved derivatives, which are either 0 or 1.

  This simplifies optimisation [ZLLS23, Sec. 5.1.2.1] and mitigates the infamous vanishing gradients problem associated with traditional activation functions.

  ReLU has gained dominance since its introduction.

  ReLU is implemented by the PyTorch function ReLU.

  However, ReLU suffers from the 💀 “dying ReLU” problem during training, when some neurons stop outputting anything other than 0 [G22, Ch. 11]:

  • During training, if a neuron’s weights get updated such that the weighted sum of the neuron’s inputs is negative, the neuron will start outputting 0.
  • When this happens, the neuron is unlikely to resurrect since the gradient of the ReLU function is 0 when its input is negative.
  • In some cases, half of the neurons die, especially when a large learning rate is used.

• The leaky ReLU (LReLU) [MHN+13] is one of the earliest extensions of ReLU:

  $\varphi(x) = \text{LReLU}(x) \triangleq \max(ax,x) = \begin{cases} x & \text{if }x>0, \\ ax & \text{otherwise,} \end{cases}$

  where $0 \lt a \lt 1$ is fixed and typically set to $0.01$.

  LReLU is differentiable except at $x=0$, but by definition, $\text{LReLU}'(x)=a>0$ for $x\leq0$, thus avoiding the dying ReLU problem.

• The parametric ReLU (PReLU) [HZRS15] extends LReLU:

  $\varphi(x) = \text{PReLU}(x) \triangleq \max(ax,x) = \begin{cases} x & \text{if }x>0, \\ ax & \text{otherwise,} \end{cases}$

  where $0 \lt a \lt 1$ is a tunable parameter controlling the slope of the negative part of PReLU, and is to be learnt jointly with the model in end-to-end training.

  PReLU is implemented by the PyTorch function PReLU.

• The exponential linear unit (ELU) [CUH16] is a smooth extension of LReLU:

  $\varphi(x) = \text{ELU}(x) \triangleq \begin{cases} x & \text{if }x>0, \\ \alpha(e^x-1) & \text{otherwise,} \end{cases}$

  where $\alpha>0$ is fixed; see Fig. 1.

  ELU is implemented by the PyTorch function ELU.

• The scaled exponential linear unit or self-normalising ELU (SELU) [KUMH17] extends ELU:

  $\varphi(x) = \text{SELU}(x) \triangleq \lambda\text{ELU}(x),$

  where $\lambda>1$ ensures a slope of larger than 1 for positive inputs; see Fig. 2.

  SELU was invented for self-normalising neural networks (SNNs), which are meant to 1️⃣ be robust to perturbations, 2️⃣ not have high variance in their training errors.

  SNNs push neuron activations to zero mean and unit variance, leading to the same effect as batch normalisation, which enables robust deep learning.

  SELU is implemented by the PyTorch function SELU.

• The Gaussian error linear unit (GELU) [HG20] extends ReLU and ELU:

  $\varphi(x) = \text{GELU}(x) \triangleq x\Phi(x) = \dfrac{x}{2}\left[1+\text{erf}(x/\sqrt{2})\right],$

  where $\Phi(x)$ is the cumulative distribution function for the Gaussian distribution, and $\text{erf}$ is the error function $\text{erf}(x)=(2/\sqrt{\pi})\int_0^xe^{-t^2}dt$.

  Unlike most other activation functions, GELU is not convex or monotonic; the increased curvature and non-monotonicity may allow GELUs to more easily approximate complicated functions than ReLUs or ELUs can.

  ReLU gates the input depending upon its sign, whereas GELU weights its input depending upon how much greater it is than other inputs.

  GELU is a popular choice for implementing transformers; see for example Hugging Face’s implementation of activation functions.

  GELU is implemented by the PyTorch function GELU.

The convolutional neural network (ConvNet or CNN) is an evolution of the multilayer perceptron that replaces matrix multiplication with convolution in at least one layer^{[GBC16, Ch. 9]}.

A deep CNN is a CNN that has more than three hidden layers.

CNNs play an important role in the history of deep learning because^{[GBC16, §9.11]}:

• They exemplify successful applications of neuroscientific insights^[Lin21] to machine learning.
• CNNs are among the first neural networks to achieve commercial success; for example, AT&T used the LeNet-5 CNN to read bank checks in the 1990s^[LBBH98].
• Deep CNNs are among the first to be successfully trained with backpropagation.
• Deep CNNs are among the first deep neural networks to achieve state-of-the-art results for challenging problems; for example, the ground-breaking performance of the 8-layer CNN called AlexNet in the 2012 ImageNet challenge^[KSH17] is widely considered to be the watershed event that propelled deep learning research.
• Deep CNNs remain pertinent for contemporary applications; for example, a combination of CNN and LSTM has been applied to predicting aircraft trajectory for assisting air crews in their decision-making during the approach phase of a flight^[LPM21].

The following discusses the basic CNN structure and zooms in on the structural elements.

Structure

Invariance to local translation is useful if detecting the presence of a feature is more important than localizing the feature.

When it was introduced, the CNN brought three architectural innovations to achieve shift/translation-invariance^{[LB02, GBC16]}:

Structural element: convolution

It has to be emphasised that the convolution in this context is inspired by but not equivalent to the convolution in linear system theory and furthermore it exists in several variants^{[GBC16, Ch. 9]}, but it is still a linear operation.

In signal processing, if $x(t)$ denotes a time-dependent input and $f(t)$ denotes the impulse response function of a linear system, then the output response of the system is given by the convolution (denoted by symbol $\ast$) of $x(t)$ and $f(t)$:

$(x\ast f)(t) = \int_{\tau=0}^t x(\tau)f(t-\tau)d\tau.$

In discrete time, the equation above can be written as

$(x\ast f)[t] = \sum_{\tau=0}^t x[t]\cdot f[t-\tau].$

Above, square brackets are used to distinguish discrete time from continuous time.

In machine learning (ML), $f(t)$ is called a kernel or filter and the output of convolution is an example of a feature map.

For two-dimensional (2D) inputs (e.g., images), we use 2D kernels in convolution:

Convolution works similarly to windowed (Gabor) Fourier transforms^{[BK19, §6.5]} and wavelet transforms^[Mal16].

Convolution is not to be confused with cross-correlation, but for efficiency, convolution is often implemented as cross-correlation in ML libraries. For example, PyTorch implements 2D convolution (Conv2d) as cross-correlation (denoted by symbol $\star$):

Note above, the indices of $f$ are $a$ and $b$ instead of $i-a$ and $j-b$. The absence of flipping of the indices makes cross-correlation more efficient.

Structural element: pooling

A pooling function replaces the output of a neural network at a certain location with a summary statistic (e.g., maximum, average) of the nearby outputs^{[GBC16, §9.3]}. The number of these nearby outputs is the pool width. Like a convolution/cross-correlation filter, a pooling window is slid over the input one stride at a time.

Fig. 5 illustrates the maximum-pooling (max-pooling for short) operation^[RP99].

Pooling helps make a feature map approximately invariant to small translations of the input. Furthermore, for many tasks, pooling is essential for handling inputs of varying size.

Fig. 6 illustrates the role of max-pooling in downsampling.

As a summary, Fig. 7 shows the structure of an example of an early CNN^[LB02]. Note though the last layers of a modern CNN are typically fully connected layers (also known as dense layers).