A recurrent neural network (RNN) is a neural network which maps an input space of sequences to an output space of sequences in a stateful way^{[RHW86, Mur22]}.

While convolutional neural networks excel at two-dimensional (2D) data, recurrent neural networks (RNNs) are better suited for one-dimensional (1D), sequential data^{[GBC16, §9.11]}.

Unlike early artificial neural networks (ANNs) which have a feedforward structure, RNNs have a cyclic structure, inspired by the cyclical connectivity of neurons; see Fig. 1.

The forward pass of an RNN is the same as that of a multilayer perceptron, except that activations arrive at a hidden layer from both the current external input and the hidden-layer activations from the previous timestep.

Fig. 1 visualises the operation of an RNN by “unfolding” or “unrolling” the network across timesteps, with the same network parameters applied at each timestep.

Note: The term “timestep” should be understood more generally as an index for sequential data.

For the backward pass, two well-known algorithms are applicable: 1️⃣ real-time recurrent learning and the simpler, computationally more efficient 2️⃣ backpropagation through time^[Wer90].

Fig. 1 implies information flows in one direction, the direction associated with causality.

However, for many sequence labelling tasks, the correct output depends on the entire input sequence, or at least a sufficiently long input sequence. Examples of these tasks include speech recognition and language translation. Addressing the need of these tasks gave rise to bidirectional RNNs^[SP97].

Standard/traditional RNNs suffer from the following deficiencies^{[Gra12, YSHZ19, MSO24]}:

• They are susceptible to the problems of vanishing gradients and exploding gradients.
• They cannot store information for long periods of time.
• Except for bidirectional RNNs, they access context information in only one direction (i.e., typically past information in the time domain).

Due to the drawbacks above, RNNs are typically used with “leaky” units enabling the networks to accumulate information over a long duration^{[GBC16, §10.10]}. The resultant RNNs are called gated RNNs. The most successful gated RNNs are those using long short-term memory (LSTM) or gated recurrent units (GRU).

Work in progress

Reinforcement learning (RL) is a family of algorithms that learn an optimal policy, whose goals is to maximize the expected return when interacting with an environment^[Goo25].

RL has existed since the 1950s^[BD10], but it was the introduction of high-capacity function approximators, namely deep neural networks, that rejuvenated RL in recent years^[LKTF20].

There are three main types of RL^{[LKTV20, FPMC24]}:

1. Online or on-policy RL: In this classic setting, an agent interacts freely with
2. Off-policy RL: In this classic setting, an agent
3. Offline RL: