Transformer (machine learning)

The transformer processes sequences in parallel, unlike recurrent networks, using a stack of encoder and decoder layers. Each layer contains multi-head self-attention and feed-forward networks with residual connections and layer normalization. Self-attention computes queries, keys, and values from input embeddings, producing weighted sums that capture long-range dependencies. Positional encodings inject order information because the model has no inherent sequence order. The original paper used sinusoidal functions, but learned positional embeddings are common in later variants. Multi-head attention runs several attention computations in parallel, allowing the model to focus on different representation subspaces. This design enables efficient training on GPUs and scales to very large datasets, which is a key advantage over recurrent architectures.

Transformers revolutionized NLP by enabling pretraining on massive corpora and fine-tuning for downstream tasks. Models like BERT (bidirectional encoder) and GPT (autoregressive decoder) set new benchmarks in question answering, translation, and text generation. The architecture's scalability led to large language models with hundreds of billions of parameters, such as GPT-3 and PaLM. These models exhibit emergent abilities like few-shot learning and chain-of-thought reasoning. Transformers also dominate machine translation, replacing earlier recurrent and convolutional approaches. The attention mechanism allows handling of long contexts, though quadratic complexity remains a challenge. Variants like sparse attention and linear attention address efficiency, enabling longer sequences.

Transformers are not limited to text; they are applied to images, audio, and multimodal data. Vision Transformer (ViT) treats image patches as tokens and achieves competitive results on image classification. In computer vision, transformers are used for object detection (DETR) and video understanding. In speech, models like Whisper and SpeechT5 use transformer architectures for recognition and synthesis. Multimodal models such as CLIP and Flamingo combine text and image encoders. Transformers also appear in reinforcement learning (Decision Transformer) and time-series forecasting. Their flexibility stems from the attention mechanism's ability to relate arbitrary positions, making them a general-purpose sequence model.

Beyond the mainstream, transformers have niche applications and historical nuances. The original paper's authors include eight researchers from Google Brain, and the name 'transformer' was chosen because the architecture transforms inputs to outputs. Early work on attention mechanisms, such as Bahdanau attention, laid the groundwork. Transformers have been used for protein structure prediction (AlphaFold 2) and drug discovery. In music generation, models like Music Transformer generate coherent compositions. Efficiency research includes Reformer (LSH attention) and Performer (kernel-based). There are also theoretical studies on the expressive power of transformers, linking them to circuit complexity. The architecture's inductive biases are still debated, with ongoing research into why they generalize so well.

Self-attention

A mechanism that computes weighted sums of input elements based on their pairwise relevance.

Multi-head attention

Multiple attention computations run in parallel, each learning different relationships.

Positional encoding

Vectors added to input embeddings to provide sequence order information.

Encoder-decoder

The two-stack structure of the original transformer, used for sequence-to-sequence tasks.

Pretraining

Training a model on a large unlabeled corpus before fine-tuning on specific tasks.