Transformer

The Transformer represents a sequence by allowing each token to weigh other tokens directly through self-attention. For a set of queries, keys, and values, attention computes compatibility scores, normalizes them, and forms weighted combinations; multiple attention heads perform this operation in parallel across different learned projections.¹

The original design paired an encoder with a decoder. Encoder layers build contextual representations, while decoder layers generate outputs one token at a time and use cross-attention to consult the encoded input. Residual connections, layer normalization, and position-wise feed-forward networks stabilize and enrich each layer. Because attention alone does not inherently encode order, the original model added sinusoidal positional encodings; later systems commonly learned positional representations or used relative-position methods.

Transformers are usually trained with large-scale self-supervision before being adapted to specific tasks. In masked-language modeling, exemplified by BERT, selected tokens are hidden and the network predicts them from both directions of context.² In autoregressive language modeling, the model predicts the next token, as in GPT-style systems.

Different descendants emphasize different parts of the original architecture. Encoder-only models are strong at classification and representation learning; decoder-only models are suited to open-ended generation; encoder–decoder models such as T5 cast many language tasks as text-to-text conversion.^{3 4 Pretraining followed by fine-tuning, instruction tuning, or preference optimization has made the architecture central to contemporary natural-language processing.}

The standard self-attention operation has quadratic computational and memory cost in sequence length, making very long contexts expensive. This limitation has motivated sparse attention, local or sliding-window attention, linear-attention approximations, retrieval augmentation, token pruning, and more efficient attention kernels.

Transformers can scale effectively with data and model size, but scale does not guarantee factual accuracy, interpretability, or robust reasoning. They may reproduce biases in training data, generate plausible false statements, and require substantial energy and specialized hardware during training. Beyond text, the Vision Transformer applies the same general idea to sequences of image patches and can compete with convolutional models when trained at suitable scale.⁵ Related adaptations handle audio, protein sequences, video, and combined image–text inputs.

The original Transformer was designed for machine translation, not for conversational text generation. Its elimination of recurrence made training more parallelizable than recurrent sequence models, while retaining the ability to connect distant positions directly.¹

Attention weights are useful diagnostic signals, but they should not automatically be treated as faithful explanations of a model's decisions. A further subtlety is that the architecture's apparent flexibility comes from several interacting components: tokenization, positional representation, normalization, initialization, optimization, and data mixture can materially affect behavior. Encoder–decoder Transformers remain important in translation and structured generation even though decoder-only models dominate public discussion of large language models. Research also explores compact and specialized Transformers for edge devices, where memory bandwidth and latency can matter more than parameter count.

Self-attention

An operation in which each sequence element forms a weighted combination of other elements according to learned compatibility scores.

Attention head

One parallel attention computation using its own learned projections of queries, keys, and values.

Encoder

A stack of layers that converts an input sequence into contextual representations.

Decoder

A stack of layers that generates an output sequence, typically with causal masking and, in the original design, cross-attention.

Positional encoding

Information added to token representations to distinguish sequence order.