Recurrent neural network

A recurrent neural network processes a sequence one element at a time while updating a hidden state that summarizes preceding elements. At step t, the network combines the current input with its previous hidden state, produces an output, and passes a new state to the next step. The same parameters are reused across positions, allowing one model to handle sequences of different lengths and making temporal order meaningful.¹

The state is not a permanent record of the entire past: it is a learned, compressed representation whose usefulness depends on the task and architecture. A basic recurrent unit commonly applies a nonlinear activation such as tanh, while outputs may be produced at every step, only at the end, or through an encoder–decoder arrangement. Bidirectional networks read a sequence in both directions when future context is available during inference.

Recurrent neural networks are typically trained with backpropagation through time, which unfolds the recurrence across the sequence and propagates errors through the resulting computational graph. Long sequences can cause vanishing gradients, which make early events difficult to learn, or exploding gradients, which can destabilize optimization. Gradient clipping, careful initialization, truncated unfolding, and normalization are practical responses, but they do not remove the underlying trade-off between memory and computation.

Long short-term memory networks address this problem with a cell state and learned input, forget, and output gates that regulate information flow.² Gated recurrent units use a more compact gating design and often offer a useful speed–accuracy compromise.³ Comparative studies show that performance depends strongly on the dataset, sequence length, regularization, and tuning rather than on one universally best recurrent cell.

Recurrent neural networks are suited to tasks in which observations arrive in order and prior context affects interpretation. They have been used for speech recognition, language modeling, machine translation, handwriting recognition, anomaly detection, and forecasting from sensors or other time-series data. Encoder–decoder recurrent models helped establish neural sequence-to-sequence translation before attention-based architectures became dominant.⁵

Their principal strengths are parameter sharing across time, natural handling of variable-length sequences, and the ability to operate incrementally as new data arrive. Their sequential dependence also limits parallelism during training and can make long-range dependencies difficult to retain. Transformer models often train more efficiently on large datasets because they process sequence positions in parallel, although recurrent models remain useful for streaming, low-latency, memory-constrained, and continuously observed settings.

Recurrent computation is broader than text generation: it can model continuous signals, irregular operational measurements, and event streams whose meaning depends on elapsed context. A recurrent network may emit a prediction at each time step, maintain state across batches in a carefully managed online system, or reset state at known sequence boundaries; confusing these choices can leak information between examples or degrade forecasts.

Recurrent networks also appear in hybrid systems rather than as standalone predictors. Convolutional layers can extract local features before a recurrent layer models their order, while attention can let a recurrent decoder retrieve particular earlier states instead of relying only on its compressed hidden state. Reservoir computing is a related approach in which a fixed recurrent dynamical network supplies rich states and only a readout is trained. These designs highlight that recurrence is a general strategy for representing evolving state, not a single fixed cell or training recipe.

Hidden state

A learned vector carried from one sequence position to the next, representing context from earlier inputs.

Backpropagation through time

A training method that unfolds recurrent computation across sequence steps before applying gradient descent.

Long short-term memory

A recurrent architecture with a cell state and gates designed to preserve or discard information over long intervals.

Gated recurrent unit

A gated recurrent architecture that combines recurrent-state updates with a relatively compact design.

Teacher forcing

A sequence-training method that supplies the target from the preceding step as the next input, rather than the model's own previous prediction.