Other meanings of Q-learning
Artificial intelligence
Q-learning is a model-free, off-policy reinforcement learning algorithm that learns the optimal action-selection policy for a given finite Markov decision process (MDP). It does so by iteratively estimating the optimal action-value function Q(s,a), which represents the expected cumulative reward of taking action a in state s and thereafter following the optimal policy. The algorithm was introduced by Christopher Watkins in his 1989 PhD thesis and has since become a foundational technique in reinforcement learning, underpinning successes in game playing, robotics, and control.
Q-learning is characterized by its off-policy learning: it estimates the optimal action-value function independently of the agent's current policy, using the Bellman optimality equation as the update rule. The update for a transition (s, a, r, s') is Q(s,a) ← Q(s,a) + α [r + γ maxa' Q(s',a') – Q(s,a)], where α is the learning rate and γ the discount factor.1 This single-step update uses the maximum over actions in the next state, making it independent of the policy used to generate the data. The algorithm converges to the optimal Q* under mild conditions (finite MDP, all state–action pairs visited infinitely often, and appropriate learning rate decay).2 Because it is off-policy, Q-learning can learn from exploratory or even suboptimal policies, a key advantage over on-policy methods like SARSA.
Watkins and Dayan (1992) proved that Q-learning converges with probability 1 to the optimal action-value function for finite MDPs under the Robbins–Monro conditions for stochastic approximation.3 The proof relies on the contraction property of the Bellman optimality operator and the fact that each Q-value update is a sample average of a fixed-point iteration. The algorithm does not require a model of the environment, and its convergence does not depend on the policy being greedy—only that every state–action pair is updated infinitely often. This robustness has made Q-learning a standard benchmark in reinforcement learning theory. However, the convergence guarantee does not extend to function approximation without careful tuning, as seen in the deadly triad of off-policy learning, function approximation, and bootstrapping.4
Q-learning has been applied to diverse domains including robotics, game playing, elevator scheduling, and finance. The most famous modern extension is the Deep Q-Network (DQN), which combines Q-learning with deep neural networks to handle high-dimensional state spaces, achieving human-level performance on Atari 2600 games.5 Other notable extensions include Double Q-learning to reduce overoptimism bias,6 Prioritized Experience Replay, and Dueling DQN architectures. In robotics, Q-learning enables online learning of manipulation and locomotion tasks without a model. Despite its successes, tabular Q-learning scales poorly with state space size, and function approximation introduces instability, motivating research into batch reinforcement learning and distributional Q-learning.
A lesser-known fact is that Q-learning's update rule is closely related to the Bellman equation originally used in dynamic programming, but its off-policy nature was a breakthrough for model-free learning. The algorithm can also be used in a continuous state space via tile coding or Fourier basis functions, yet convergence guarantees become fragile. Another subtlety: Q-learning is equivalent to solving the optimality equation for the optimal action-value function, but it does not require the policy to be deterministic. There is also a popular misconception that Q-learning is always off-policy—it can be used on-policy if the behavioural policy is the same as the target policy, but it is typically employed off-policy. Double Q-learning (Hasselt, 2010) was developed specifically to address the maximization bias in the standard Q-learning update, a problem that can cause significant performance degradation in stochastic environments.6 Moreover, the earliest form of Q-learning appeared in Watkins's 1989 thesis but was initially little noticed outside the reinforcement learning community until the 1990s.
Help improve the encyclopedia. Reports go straight to the site manager.