Reinforcement learning

Reinforcement learning treats decision-making as a sequence of actions whose consequences may arrive immediately or much later. An agent observes a state, selects an action, receives a reward, and moves to another state; its objective is to maximize expected cumulative reward rather than merely the next payoff. The mathematical framework is commonly expressed as a Markov decision process, which specifies states, actions, transition probabilities, and rewards.

The field draws on several intellectual traditions, including dynamic programming, optimal control, animal-learning psychology, and trial-and-error learning. Richard Bellman’s work on the principle of optimality supplied a central mathematical foundation, while temporal-difference learning later provided a way to update predictions from successive experience without waiting for an episode to end. Unlike supervised learning, reinforcement learning does not require a correct action label for each situation; unlike unsupervised learning, it is organized around goals and evaluative feedback.

Agents learn by estimating which actions are valuable, which policies are effective, or both. A value function estimates the expected future return from a state or state–action pair, while a policy specifies how the agent chooses actions. The return may be discounted, giving near-term rewards greater weight than distant ones.

Learning requires balancing exploration and exploitation. Exploration gathers information about unfamiliar actions, whereas exploitation selects actions already believed to be effective. Temporal-difference methods update estimates after each transition; Q-learning, for example, learns an action-value function using a target based on the best estimated next action, while remaining off-policy because its learned target need not match the behavior used to collect data.¹

Policy-gradient methods instead adjust policy parameters directly in the direction that increases expected return. Actor–critic systems combine a policy-producing actor with a value-estimating critic, a design that supports learning in large or continuous action spaces.

Modern reinforcement learning combines classical value estimation with function approximation, especially neural networks. Deep Q-networks made a notable advance by learning from raw pixels while using experience replay and a separate target network to improve training stability; a 2015 study reported strong performance across a broad set of Atari 2600 games.²

Other important families include Monte Carlo methods, policy optimization, model-based planning, and multi-agent learning. Model-free methods learn behavior without constructing an explicit environment model, while model-based methods learn or use a model to simulate possible futures. The distinction is practical rather than absolute: contemporary systems often combine learned models, value functions, and policies.

Deep reinforcement learning remains difficult because neural networks can be unstable when data are correlated, rewards are sparse, or the environment changes. Sample efficiency, reliable exploration, and generalization beyond training conditions remain central research problems.

Reinforcement learning is especially suited to sequential tasks in which actions affect later opportunities, including robotics, resource allocation, recommendation, games, and industrial control. Its clearest public demonstrations have come from games, where rules provide a measurable environment and outcomes can be evaluated repeatedly.

DeepMind’s AlphaGo combined supervised learning from human games with reinforcement learning through self-play and defeated leading professional Go player Lee Sedol in 2016.³ AlphaZero later showed that a related system could learn strong play in chess, shogi, and Go primarily from the rules and self-play rather than human game records.⁴ MuZero extended this direction by learning an internal model useful for planning without being given the environment’s full rules in an explicit form.⁵ In physical applications, safety constraints and costly experimentation make purely trial-and-error learning impractical, so simulation, demonstrations, offline data, and human-designed safeguards are often combined with reinforcement learning.

Reinforcement learning can fail even when its optimization process is working correctly, because the specified reward may not capture the designer’s real intention. An agent can exploit loopholes, maximize an easily measured proxy, or find an unexpected strategy that scores well while violating the broader goal; this phenomenon is commonly called specification gaming.⁶

Many real environments are partially observable: the agent cannot see the complete state, so it must infer hidden conditions from a history of observations. Delayed rewards create a related credit-assignment problem, making it difficult to determine which earlier action caused a later outcome. Multi-agent settings add competition, cooperation, and nonstationarity, because other agents may change their behavior while learning.

Reinforcement learning also has a substantial offline variant, in which an agent learns from previously collected records rather than by interacting freely with the environment. This is valuable in medicine, robotics, and autonomous systems, where unsafe exploratory actions are unacceptable, but it introduces distribution-shift risks when the learned policy proposes actions poorly represented in the data.

Agent

The learner or decision-maker that selects actions.

Environment

The external system in which the agent acts and from which it receives observations and rewards.

Policy

A rule or probability distribution specifying which action an agent selects in a given situation.

Return

The cumulative, often discounted, reward used as the objective for learning.

Value function

An estimate of expected future return from a state or state–action pair.

Exploration–exploitation trade-off

The tension between trying uncertain actions and choosing actions already believed to perform well.