This knowledge base entry follows discussion of artificial neural networks and backpropagation.

Contemporary options for $\varphi$ are the non-saturating activation functions [Mur22, Sec. 13.4.3], although the term is not accurate.

Below, ($x$ should be understood as the output of the summing junction.

• The rectified linear unit (ReLU) [NH10] is the unipolar function:

  $\varphi(x) = \text{ReLU}(x) \triangleq \max(0,x) = \begin{cases} x & \text{if }x>0, \\ 0 & \text{otherwise.} \end{cases}$

  ReLU is differentiable except at $x=0$, but by definition, $\text{ReLU}'(x)=0$ for $x\leq0$.

  ReLU has the advantage of having well-behaved derivatives, which are either 0 or 1.

  This simplifies optimisation [ZLLS23, Sec. 5.1.2.1] and mitigates the infamous vanishing gradients problem associated with traditional activation functions.

  ReLU has gained dominance since its introduction.

  ReLU is implemented by the PyTorch function ReLU.

  However, ReLU suffers from the 💀 “dying ReLU” problem during training, when some neurons stop outputting anything other than 0 [G22, Ch. 11]:

  • During training, if a neuron’s weights get updated such that the weighted sum of the neuron’s inputs is negative, the neuron will start outputting 0.
  • When this happens, the neuron is unlikely to resurrect since the gradient of the ReLU function is 0 when its input is negative.
  • In some cases, half of the neurons die, especially when a large learning rate is used.

• The leaky ReLU (LReLU) [MHN+13] is one of the earliest extensions of ReLU:

  $\varphi(x) = \text{LReLU}(x) \triangleq \max(ax,x) = \begin{cases} x & \text{if }x>0, \\ ax & \text{otherwise,} \end{cases}$

  where $0 \lt a \lt 1$ is fixed and typically set to $0.01$.

  LReLU is differentiable except at $x=0$, but by definition, $\text{LReLU}'(x)=a>0$ for $x\leq0$, thus avoiding the dying ReLU problem.

• The parametric ReLU (PReLU) [HZRS15] extends LReLU:

  $\varphi(x) = \text{PReLU}(x) \triangleq \max(ax,x) = \begin{cases} x & \text{if }x>0, \\ ax & \text{otherwise,} \end{cases}$

  where $0 \lt a \lt 1$ is a tunable parameter controlling the slope of the negative part of PReLU, and is to be learnt jointly with the model in end-to-end training.

  PReLU is implemented by the PyTorch function PReLU.

• The exponential linear unit (ELU) [CUH16] is a smooth extension of LReLU:

  $\varphi(x) = \text{ELU}(x) \triangleq \begin{cases} x & \text{if }x>0, \\ \alpha(e^x-1) & \text{otherwise,} \end{cases}$

  where $\alpha>0$ is fixed; see Fig. 1.

  ELU is implemented by the PyTorch function ELU.

• The scaled exponential linear unit or self-normalising ELU (SELU) [KUMH17] extends ELU:

  $\varphi(x) = \text{SELU}(x) \triangleq \lambda\text{ELU}(x),$

  where $\lambda>1$ ensures a slope of larger than 1 for positive inputs; see Fig. 2.

  SELU was invented for self-normalising neural networks (SNNs), which are meant to 1️⃣ be robust to perturbations, 2️⃣ not have high variance in their training errors.

  SNNs push neuron activations to zero mean and unit variance, leading to the same effect as batch normalisation, which enables robust deep learning.

  SELU is implemented by the PyTorch function SELU.

• The Gaussian error linear unit (GELU) [HG20] extends ReLU and ELU:

  $\varphi(x) = \text{GELU}(x) \triangleq x\Phi(x) = \dfrac{x}{2}\left[1+\text{erf}(x/\sqrt{2})\right],$

  where $\Phi(x)$ is the cumulative distribution function for the Gaussian distribution, and $\text{erf}$ is the error function $\text{erf}(x)=(2/\sqrt{\pi})\int_0^xe^{-t^2}dt$.

  Unlike most other activation functions, GELU is not convex or monotonic; the increased curvature and non-monotonicity may allow GELUs to more easily approximate complicated functions than ReLUs or ELUs can.

  ReLU gates the input depending upon its sign, whereas GELU weights its input depending upon how much greater it is than other inputs.

  GELU is a popular choice for implementing transformers; see for example Hugging Face’s implementation of activation functions.

  GELU is implemented by the PyTorch function GELU.