Transformer

The Transformer represents a sequence as vectors and repeatedly refines them through attention and feed-forward computation. In self-attention, each token produces a query, key, and value; similarities between queries and keys determine how strongly each value contributes to the token’s updated representation.¹ Multi-head attention runs several learned attention patterns in parallel, allowing different heads to capture relationships such as syntax, position, or repetition. Since attention alone does not encode order, the model receives positional information, either through fixed positional encodings or learned and relative position schemes. Each block normally also contains residual connections, layer normalization, and a position-wise multilayer perceptron.

Unlike a recurrent neural network, the architecture processes all training positions in parallel. Its standard full-attention operation, however, compares every pair of positions, giving it quadratic memory and computation in sequence length. This makes long-context inference a central engineering problem.

Transformer behavior depends strongly on how its attention blocks are arranged and trained. The original design paired an encoder, which reads an entire input, with a decoder, which generates an output while masking future positions.¹ Encoder-only models such as BERT learn context-sensitive representations through masked-token prediction and are commonly adapted for classification, retrieval, and token labeling.² Decoder-only models predict the next token from preceding context and support open-ended generation; GPT-3 demonstrated how large-scale autoregressive training could produce broad few-shot capabilities.³

Training usually begins with self-supervised pretraining on large corpora, followed by fine-tuning, prompting, or preference-based alignment. Tokenization, data mixture, context length, parameter count, optimization, and evaluation all shape the resulting capabilities and failure modes; size alone is not a complete explanation.

Transformers have become general-purpose sequence models because the same basic operations can handle text, images, speech, code, and biological sequences. Vision Transformer showed that an image can be divided into patches and processed as a sequence, with competitive image-classification results when trained at suitable scale.⁴ Related systems translate languages, transcribe speech, generate code, retrieve information, model proteins, and combine several modalities.

The architecture does not guarantee factual accuracy, understanding, or harmless behavior. Models can reproduce biases in data, generate fabricated statements, leak memorized information, and be sensitive to wording. Full attention becomes expensive for long documents, while autoregressive decoding remains sequential even when training is parallel. Deployment therefore uses compression, caching, retrieval, sparse or local attention, and specialized hardware, with evaluation needed for the particular domain and stakes.

Several less visible design choices have had outsized effects on Transformer systems. The original architecture replaced recurrent computation with attention, but later work showed that efficient kernels can substantially reduce the practical cost of attention without changing its mathematical result; FlashAttention does this by reorganizing memory access and tiling computation.⁵ Relative position methods, rotary representations, grouped-query attention, and sliding-window patterns address different compromises among context, speed, and quality.

Scaling is also not simply a matter of adding parameters. Empirical scaling-law research found predictable relationships among model size, data, and compute, while later training practice emphasized balancing these quantities rather than leaving a large model undertrained.⁶ Small Transformers can be effective when distillation, task-specific data, retrieval, or restricted vocabularies reduce the problem’s scope.

Self-attention

An operation in which each sequence element weights information from other elements using learned query, key, and value vectors.

Token

A discrete unit, such as a word fragment, character, image patch, or code symbol, represented numerically by a model.

Encoder

A Transformer component that builds representations from an input sequence, generally allowing interaction among all input positions.

Decoder

A Transformer component designed to produce outputs, often with a causal mask that prevents access to future positions.

Causal language modeling

Training a model to predict each next token from the tokens that precede it.

Context window

The maximum sequence of tokens that a model can process together in one input context.