Large language model

Large language models are built on the transformer architecture, introduced in 2017 by Vaswani et al.¹ This architecture uses a self-attention mechanism to weigh the importance of different tokens in a sequence, enabling parallel processing and long-range dependencies. Training involves predicting the next token in a sequence over vast corpora—often hundreds of billions of words from the internet, books, and other sources.² The scale of parameters (weights) has grown from millions in early models to trillions; GPT-3 (175 billion parameters) and GPT-4 (estimated 1.8 trillion) exemplify this trend.³ Emergent abilities—such as few-shot learning, reasoning, and code generation—appear only at large scales, following empirical scaling laws.⁴

LLMs are typically pretrained in an unsupervised or self-supervised manner on unlabeled text, then fine-tuned on labeled datasets for specific tasks. Reinforcement Learning from Human Feedback (RLHF) is used to align outputs with human preferences, reducing harmful or biased responses.³ Capabilities extend beyond text generation to translation, summarization, question answering, and even mathematical reasoning. However, models suffer from hallucination—generating plausible but incorrect facts—and inherit biases present in their training data.⁵ Context windows, which limit the amount of text the model can process simultaneously, have expanded from 2,048 tokens (GPT-2) to 128,000 tokens (GPT-4 Turbo).³

Early precursors include rule-based systems like ELIZA (1966) and statistical language models, but the modern LLM lineage began with the transformer. The term "stochastic parrots" was coined by Bender et al. (2021) to critique the risk of models merely recombining patterns without understanding.⁵ Scaling laws, derived by Kaplan et al. (2020), show that performance improves predictably with model size, dataset size, and compute—a finding that has driven the race toward ever-larger models.⁴ Training a single LLM can emit as much CO₂ as several cars over their lifetimes, raising environmental concerns.⁶ LLMs are also used for scientific discovery, such as generating candidate drug molecules or predicting protein structures (e.g., AlphaFold is not an LLM, but LLMs assist in literature mining).

LLMs power chatbots (ChatGPT, Gemini), code assistants (GitHub Copilot), and search engines, transforming user interaction with technology. In medicine, they assist with clinical note summarization and differential diagnosis; in law, they draft contracts and analyze case law.⁷ Educational tools use LLMs to answer student questions and generate exercises. However, widespread deployment raises concerns about job displacement, misinformation, and the centralization of AI power in a few corporations. Regulatory efforts, such as the EU AI Act, classify LLMs as "general-purpose AI systems" subject to transparency obligations.⁸

Transformer

A neural network architecture using self-attention, introduced in 2017, that forms the basis of most modern LLMs.

Pretraining

Initial training phase on a large, unlabeled corpus to learn general language features.

Fine-tuning

Subsequent training on a smaller labeled dataset to adapt the model to a specific task.

Token

A unit of text (word, subword, or character) that the model processes.

Attention mechanism

A component that allows the model to focus on different parts of the input sequence when generating each output token.