This knowledge base entry follows discussion of artificial neural networks and backpropagation.

Contemporary options for $\varphi$ are the non-saturating activation functions [Mur22, Sec. 13.4.3], although the term is not accurate.

Below, ($x$ should be understood as the output of the summing junction.

• The rectified linear unit (ReLU) [NH10] is the unipolar function:

  $\varphi(x) = \text{ReLU}(x) \triangleq \max(0,x) = \begin{cases} x & \text{if }x>0, \\ 0 & \text{otherwise.} \end{cases}$

  ReLU is differentiable except at $x=0$, but by definition, $\text{ReLU}'(x)=0$ for $x\leq0$.

  ReLU has the advantage of having well-behaved derivatives, which are either 0 or 1.

  This simplifies optimisation [ZLLS23, Sec. 5.1.2.1] and mitigates the infamous vanishing gradients problem associated with traditional activation functions.

  ReLU has gained dominance since its introduction.

  ReLU is implemented by the PyTorch function ReLU.

  However, ReLU suffers from the 💀 “dying ReLU” problem during training, when some neurons stop outputting anything other than 0 [G22, Ch. 11]:

  • During training, if a neuron’s weights get updated such that the weighted sum of the neuron’s inputs is negative, the neuron will start outputting 0.
  • When this happens, the neuron is unlikely to resurrect since the gradient of the ReLU function is 0 when its input is negative.
  • In some cases, half of the neurons die, especially when a large learning rate is used.

• The leaky ReLU (LReLU) [MHN+13] is one of the earliest extensions of ReLU:

  $\varphi(x) = \text{LReLU}(x) \triangleq \max(ax,x) = \begin{cases} x & \text{if }x>0, \\ ax & \text{otherwise,} \end{cases}$

  where $0 \lt a \lt 1$ is fixed and typically set to $0.01$.

  LReLU is differentiable except at $x=0$, but by definition, $\text{LReLU}'(x)=a>0$ for $x\leq0$, thus avoiding the dying ReLU problem.

• The parametric ReLU (PReLU) [HZRS15] extends LReLU:

  $\varphi(x) = \text{PReLU}(x) \triangleq \max(ax,x) = \begin{cases} x & \text{if }x>0, \\ ax & \text{otherwise,} \end{cases}$

  where $0 \lt a \lt 1$ is a tunable parameter controlling the slope of the negative part of PReLU, and is to be learnt jointly with the model in end-to-end training.

  PReLU is implemented by the PyTorch function PReLU.

• The exponential linear unit (ELU) [CUH16] is a smooth extension of LReLU:

  $\varphi(x) = \text{ELU}(x) \triangleq \begin{cases} x & \text{if }x>0, \\ \alpha(e^x-1) & \text{otherwise,} \end{cases}$

  where $\alpha>0$ is fixed; see Fig. 1.

  ELU is implemented by the PyTorch function ELU.

• The scaled exponential linear unit or self-normalising ELU (SELU) [KUMH17] extends ELU:

  $\varphi(x) = \text{SELU}(x) \triangleq \lambda\text{ELU}(x),$

  where $\lambda>1$ ensures a slope of larger than 1 for positive inputs; see Fig. 2.

  SELU was invented for self-normalising neural networks (SNNs), which are meant to 1️⃣ be robust to perturbations, 2️⃣ not have high variance in their training errors.

  SNNs push neuron activations to zero mean and unit variance, leading to the same effect as batch normalisation, which enables robust deep learning.

  SELU is implemented by the PyTorch function SELU.

• The Gaussian error linear unit (GELU) [HG20] extends ReLU and ELU:

  $\varphi(x) = \text{GELU}(x) \triangleq x\Phi(x) = \dfrac{x}{2}\left[1+\text{erf}(x/\sqrt{2})\right],$

  where $\Phi(x)$ is the cumulative distribution function for the Gaussian distribution, and $\text{erf}$ is the error function $\text{erf}(x)=(2/\sqrt{\pi})\int_0^xe^{-t^2}dt$.

  Unlike most other activation functions, GELU is not convex or monotonic; the increased curvature and non-monotonicity may allow GELUs to more easily approximate complicated functions than ReLUs or ELUs can.

  ReLU gates the input depending upon its sign, whereas GELU weights its input depending upon how much greater it is than other inputs.

  GELU is a popular choice for implementing transformers; see for example Hugging Face’s implementation of activation functions.

  GELU is implemented by the PyTorch function GELU.

This knowledge base entry follows discussion of artificial neural networks and backpropagation.

The backpropagation (“backprop” for short) algorithm calculates gradients to update each weight.

Both problems plague deep neural networks (DNNs) and recurrent neural networks (RNNs) over very long sequences [Mur22, Sec. 13.4.2].

More generally, deep neural networks suffer from unstable gradients, and different layers may learn at widely different speeds.

Watch Prof Ng’s explanation of the problems:

The problems were observed decades ago and were the reasons why DNNs were mostly abandoned in the early 2000s [G22, Ch. 11].

• The causes had been traced to the 1️⃣ usage of sigmoid activation functions, and 2️⃣ initialisation of weights to follow the zero-mean Gaussian distribution with standard deviation 1.
• A sigmoid function saturates at 0 or 1, and when saturated, the derivative is nearly 0.
• As a remedy, current best practices include using 1️⃣ a rectifier activation function, and 2️⃣ the weight initialisation algorithm called He initialisation.
• He initialisation [HZRS15, Sec. 2.2]: at layer $\ell$, weights follow the zero-mean Gaussian distribution with variance $2/n_\ell$, where $n_\ell$ is the fan-in, or equivalently the number of inputs/weights feeding into layer $\ell$.
• He initialisation is implemented by the PyTorch function kaiming_normal_ and the Tensorflow function HeNormal.

Watch Prof Ng’s explanation of weight initialisation: