This knowledge base entry follows discussion of artificial neural networks and backpropagation.

Contemporary options for $\varphi$ are the non-saturating activation functions [Mur22, Sec. 13.4.3], although the term is not accurate.

Below, $x$ should be understood as the output of the summing junction.

• The rectified linear unit (ReLU) [NH10] is the unipolar function:

  $\varphi(x) = \text{ReLU}(x) \triangleq \max(0,x) = \begin{cases} x & \text{if }x>0, \\ 0 & \text{otherwise.} \end{cases}$

  ReLU is differentiable except at $x=0$, but by definition, $\text{ReLU}'(x)=0$ for $x\leq0$.

  ReLU has the advantage of having well-behaved derivatives, which are either 0 or 1.

  This simplifies optimisation [ZLLS23, Sec. 5.1.2.1] and mitigates the infamous vanishing gradients problem associated with traditional activation functions.

  ReLU has gained dominance since its introduction.

  ReLU is implemented by the PyTorch function ReLU.

  However, ReLU suffers from the 💀 “dying ReLU” problem during training, when some neurons stop outputting anything other than 0 [G22, Ch. 11]:

  • During training, if a neuron’s weights get updated such that the weighted sum of the neuron’s inputs is negative, the neuron will start outputting 0.
  • When this happens, the neuron is unlikely to resurrect since the gradient of the ReLU function is 0 when its input is negative.
  • In some cases, half of the neurons die, especially when a large learning rate is used.

• The leaky ReLU (LReLU) [MHN+13] is one of the earliest extensions of ReLU:

  $\varphi(x) = \text{LReLU}(x) \triangleq \max(ax,x) = \begin{cases} x & \text{if }x>0, \\ ax & \text{otherwise,} \end{cases}$

  where $0 \lt a \lt 1$ is fixed and typically set to $0.01$.

  LReLU is differentiable except at $x=0$, but by definition, $\text{LReLU}'(x)=a>0$ for $x\leq0$, thus avoiding the dying ReLU problem.

• The parametric ReLU (PReLU) [HZRS15] extends LReLU:

  $\varphi(x) = \text{PReLU}(x) \triangleq \max(ax,x) = \begin{cases} x & \text{if }x>0, \\ ax & \text{otherwise,} \end{cases}$

  where $0 \lt a \lt 1$ is a tunable parameter controlling the slope of the negative part of PReLU, and is to be learnt jointly with the model in end-to-end training.

  PReLU is implemented by the PyTorch function PReLU.

• The exponential linear unit (ELU) [CUH16] is a smooth extension of LReLU:

  $\varphi(x) = \text{ELU}(x) \triangleq \begin{cases} x & \text{if }x>0, \\ \alpha(e^x-1) & \text{otherwise,} \end{cases}$

  where $\alpha>0$ is fixed; see Fig. 1.

  ELU is implemented by the PyTorch function ELU.

• The scaled exponential linear unit or self-normalising ELU (SELU) [KUMH17] extends ELU:

  $\varphi(x) = \text{SELU}(x) \triangleq \lambda\text{ELU}(x),$

  where $\lambda>1$ ensures a slope of larger than 1 for positive inputs; see Fig. 2.

  SELU was invented for self-normalising neural networks (SNNs), which are meant to 1️⃣ be robust to perturbations, 2️⃣ not have high variance in their training errors.

  SNNs push neuron activations to zero mean and unit variance, leading to the same effect as batch normalisation, which enables robust deep learning.

  SELU is implemented by the PyTorch function SELU.

• The Gaussian error linear unit (GELU) [HG20] extends ReLU and ELU:

  $\varphi(x) = \text{GELU}(x) \triangleq x\Phi(x) = \dfrac{x}{2}\left[1+\text{erf}(x/\sqrt{2})\right],$

  where $\Phi(x)$ is the cumulative distribution function for the Gaussian distribution, and $\text{erf}$ is the error function $\text{erf}(x)=(2/\sqrt{\pi})\int_0^xe^{-t^2}dt$.

  Unlike most other activation functions, GELU is not convex or monotonic; the increased curvature and non-monotonicity may allow GELUs to more easily approximate complicated functions than ReLUs or ELUs can.

  ReLU gates the input depending upon its sign, whereas GELU weights its input depending upon how much greater it is than other inputs.

  GELU is a popular choice for implementing transformers; see for example Hugging Face’s implementation of activation functions.

  GELU is implemented by the PyTorch function GELU.

The convolutional neural network (ConvNet or CNN) is an evolution of the multilayer perceptron that replaces matrix multiplication with convolution in at least one layer^{[GBC16, Ch. 9]}.

A deep CNN is a CNN that has more than three hidden layers.

CNNs play an important role in the history of deep learning because^{[GBC16, §9.11]}:

• They exemplify successful applications of neuroscientific insights^[Lin21] to machine learning.
• CNNs are among the first neural networks to achieve commercial success; for example, AT&T used the LeNet-5 CNN to read bank checks in the 1990s^[LBBH98].
• Deep CNNs are among the first to be successfully trained with backpropagation.
• Deep CNNs are among the first deep neural networks to achieve state-of-the-art results for challenging problems; for example, the ground-breaking performance of the 8-layer CNN called AlexNet in the 2012 ImageNet challenge^[KSH17] is widely considered to be the watershed event that propelled deep learning research.
• Deep CNNs remain pertinent for contemporary applications; for example, a combination of CNN and LSTM has been applied to predicting aircraft trajectory for assisting air crews in their decision-making during the approach phase of a flight^[LPM21].

The following discusses the basic CNN structure and zooms in on the structural elements.

Structure

Invariance to local translation is useful if detecting the presence of a feature is more important than localizing the feature.

When it was introduced, the CNN brought three architectural innovations to achieve shift/translation-invariance^{[LB02, GBC16]}:

Structural element: convolution

It has to be emphasised that the convolution in this context is inspired by but not equivalent to the convolution in linear system theory and furthermore it exists in several variants^{[GBC16, Ch. 9]}, but it is still a linear operation.

In signal processing, if $x(t)$ denotes a time-dependent input and $f(t)$ denotes the impulse response function of a linear system, then the output response of the system is given by the convolution (denoted by symbol $\ast$) of $x(t)$ and $f(t)$:

$(x\ast f)(t) = \int_{\tau=0}^t x(\tau)f(t-\tau)d\tau.$

In discrete time, the equation above can be written as

$(x\ast f)[t] = \sum_{\tau=0}^t x[t]\cdot f[t-\tau].$

Above, square brackets are used to distinguish discrete time from continuous time.

In machine learning (ML), $f(t)$ is called a kernel or filter and the output of convolution is an example of a feature map.

For two-dimensional (2D) inputs (e.g., images), we use 2D kernels in convolution:

Convolution works similarly to windowed (Gabor) Fourier transforms^{[BK19, §6.5]} and wavelet transforms^[Mal16].

Convolution is not to be confused with cross-correlation, but for efficiency, convolution is often implemented as cross-correlation in ML libraries. For example, PyTorch implements 2D convolution (Conv2d) as cross-correlation (denoted by symbol $\star$):

Note above, the indices of $f$ are $a$ and $b$ instead of $i-a$ and $j-b$. The absence of flipping of the indices makes cross-correlation more efficient.

Structural element: pooling

A pooling function replaces the output of a neural network at a certain location with a summary statistic (e.g., maximum, average) of the nearby outputs^{[GBC16, §9.3]}. The number of these nearby outputs is the pool width. Like a convolution/cross-correlation filter, a pooling window is slid over the input one stride at a time.

Fig. 5 illustrates the maximum-pooling (max-pooling for short) operation^[RP99].

Pooling helps make a feature map approximately invariant to small translations of the input. Furthermore, for many tasks, pooling is essential for handling inputs of varying size.

Fig. 6 illustrates the role of max-pooling in downsampling.

As a summary, Fig. 7 shows the structure of an example of an early CNN^[LB02]. Note though the last layers of a modern CNN are typically fully connected layers (also known as dense layers).

A long short-term memory (LSTM) network is a type of recurrent neural network (RNN) designed to address the problems of vanishing gradients and exploding gradients using gradient truncation and structures called “constant error carousels” for enforcing constant (as opposed to vanishing or exploding) error flow^[HS97].

LSTM solves such a fundamental problem with traditional RNNs that most of the state-of-the-art results achieved through RNNs can be attributed to LSTM^[YSHZ19].

An LSTM network replaces the traditional neural network layers with LSTM layers, each of which consists of a set of recurrently connected, differentiable memory blocks^[GS05].

Each LSTM block typically contains one recurrently connected memory cell, called an LSTM cell (to be distinguished from a neuron, which is also called a node or unit), but can contain multiple cells.

LSTM networks can be classified into two main types^{[YSHZ19, VHMN20]}:

Since the mid 2010s, advances in machine learning (ML) and particularly deep learning (DL), under the banner of artificial intelligence (AI), have been attracting not only media attention but also major capital investments.

The field of ML is decades old, but it was not until 2012, when deep neural networks (DNNs) emerged triumphant in the ImageNet image classification challenge [KSH17], that the field of ML truly took off.

DL is known to have approached or even exceeded human-level performance in many tasks.

DL techniques, especially DNN algorithms, are our main pursuit in this course, but before diving into them, we should get a clear idea about the differences among ML, DL and AI; starting with the subsequent definitions.

AI has the broadest and yet most elusive definition. There are four main schools of thought [RN22, Sec. 1.1; IBM23], namely 1️⃣ systems that think like humans, 2️⃣ systems that act like humans, 3️⃣ systems that think rationally, 4️⃣ systems that act rationally; but a sensible definition boils down to:

Above, a rational agent is one that acts so as to achieve the best outcome or, when there is uncertainty, the best expected outcome [RN22, Sec. 1.1.4].

The definition of AI above is referred to as the standard model of AI [RN22, Sec. 1.1.4].

ML is a subfield of AI:

In the preceding definition, “experience”, “task” and “performance measure” require elaboration. Among the most common ML tasks are:

• Classification: This is usually achieved through supervised learning (see Fig. 1), the aim of which is to learn a mapping $f$ from the input set $\mathcal{X}$ to the output set $\mathcal{Y}$ [ Mur22, Sec. 1.2; GBC16, Sec. 5.1.3; G22, Ch. 1], where

  • every member of $\mathcal{X}$ is a vector of features, attributes, covariates, or predictors;
  • every member of $\mathcal{Y}$ is a label, target, or response;
  • each pair of input $x\in\mathcal{X}$ and associated output $y\in\mathcal{Y}$ is called an example.

  A dataset containing $\mathcal{X}$ and $\mathcal{Y}$ used to “train” a model to predict/infer $y$ given some $x$ is called a training set; and this corresponds to experience $E$ in Definition 2.

  When $\mathcal{Y}$ is a set of unordered and mutually exclusive labels known as classes, the supervised learning task becomes a classification task.

  Classification of only two classes is called binary classification. For example, determining whether an email is spam or not is a binary classification task.

Fig. 1: Supervised learning [ZLLS23, Fig. 1.3.1].

Fig. 2: An example of a regression problem, where given a “new instance”, the target value is to be determined [G22, Figure 1-6].

• Regression: Continuing from classification, if the output set $\mathcal{Y}$ is a continuous set of real values, rather than a discrete set, the classification task becomes a regression task.

  For example, given the features (e.g., mileage, age, brand, model) and associated price for many examples of cars, a plausible regression task is to predict the price of a car given its features; see Fig. 2.

  While the term “label” is more common in the classification context, “target” is more common in the regression context. In the earlier example, the target is the car price.

• Clustering: This is the grouping of similar things together.

  The Euclidean distance between two feature vectors can serve as a similarity measure, but depending on the problem, other similarity measures can be more suitable. In fact, many similarity measures have been proposed in the literature [GMW07, Ch. 6].

  Clustering is a form of unsupervised learning.

  From a probabilistic viewpoint, unsupervised learning is fitting an unconditional model of the form $\Pr\{x\}$, which can generate new data $x$, whereas supervised learning involves fitting a conditional model, $\Pr\{y|x\}$, which specifies (a distribution over) outputs given inputs [Mur22, Sec. 1.3].

• Anomaly detection: This is another form of unsupervised learning, and highly relevant to this course of ours.

  We first encountered anomaly detection in Tutorial 1 on intrusion detection, and we will dive deep into anomaly detection in Tutorial 5 on unsupervised learning.

Common to the aforementioned tasks is the need to measure performance. An example of performance measure is

$\text{accuracy} = \frac{\text{number of correct predictions}}{\text{total number of predictions}}.$

Other performance measures will be investigated as part of Task 1.

The rest of this tutorial attempts to 1️⃣ shed some light on why DNNs are superior to classical ML algorithms, 2️⃣ provide a brief tutorial on the original/shallow/artificial neural networks (ANNs), and 3️⃣ provide a preview of DNNs.

The good news with the topics of this tutorial is that there is such a vast amount of learning resources in the public domain, that even if the coverage here fails to satisfy your learning needs, there must be some resources out there that can.

This knowledge base entry follows discussion of artificial neural networks and backpropagation.

The backpropagation (“backprop” for short) algorithm calculates gradients to update each weight.

Both problems plague deep neural networks (DNNs) and recurrent neural networks (RNNs) over very long sequences [Mur22, Sec. 13.4.2].

More generally, deep neural networks suffer from unstable gradients, and different layers may learn at widely different speeds.

Watch Prof Ng’s explanation of the problems:

The problems were observed decades ago and were the reasons why DNNs were mostly abandoned in the early 2000s [G22, Ch. 11].

• The causes had been traced to the 1️⃣ usage of sigmoid activation functions, and 2️⃣ initialisation of weights to follow the zero-mean Gaussian distribution with standard deviation 1.
• A sigmoid function saturates at 0 or 1, and when saturated, the derivative is nearly 0.
• As a remedy, current best practices include using 1️⃣ a rectifier activation function, and 2️⃣ the weight initialisation algorithm called He initialisation.
• He initialisation [HZRS15, Sec. 2.2]: at layer $\ell$, weights follow the zero-mean Gaussian distribution with variance $2/n_\ell$, where $n_\ell$ is the fan-in, or equivalently the number of inputs/weights feeding into layer $\ell$.
• He initialisation is implemented by the PyTorch function kaiming_normal_ and the Tensorflow function HeNormal.

Watch Prof Ng’s explanation of weight initialisation:

Different authors classify SSL algorithms slightly differently, but based on the pretext tasks, two distinct approaches are identifiable, namely generative and contrastive (see Fig. 2); the other approaches are either a hybrid of these two approaches, namely generative-contrastive / adversarial, or something else entirely.

Given a dataset $\mathbf{X}\in\mathbb{R}^{N\times n}$, where $N$ denotes the number of samples and $n$ denotes the number of features, it is a common practice to preprocess $\mathbf{X}$ so that each column has zero mean and unit variance; this is called standardising the data [Mur22, Sec. 7.4.5].

Standardising forces the variance (per column) to be 1 but does not remove correlation between columns.

Decorrelation necessitates whitening.

Whitening is a linear transformation of measurement $\vec{x}$ that produces decorrelated $\tilde{\vec{x}} = \mathbf{W}\vec{x}$ such that the covariance matrix $\mathbf{\Sigma} \triangleq \mathsf{E}\{\tilde{\vec{x}}\tilde{\vec{x}}^\top\} = \mathbf{I}$, where $\mathbf{W}$ is called a whitening matrix [CPSK07, Sec. 2.5.3].

All $\tilde{\vec{x}}$, $\mathbf{W}$ and $\mathbf{I}$ have the same number of rows, denoted $\ell$,  which satisfies $\ell\leq n$; if $\ell\lt n$, then dimensionality reduction is also achieved besides whitening.

A whitening matrix can be obtained using eigenvalue decomposition:

$\mathbf{\Sigma} = \mathbf{EDE}^\top.$

where $\mathbf{E}$ is an orthogonal matrix containing the covariance matrix’s eigenvectors as its columns, and $\mathbf{D}$ is the diagonal matrix of the covariance matrix’s eigenvalues [ZX09, p. 74]. Based on the decomposition, the whitening matrix can be defined as

$\mathbf{W} = \mathbf{D}^{-/2}\mathbf{E}^\top.$

$\mathbf{W}$ above is called the PCA whitening matrix [Mur22, Sec. 7.4.5].