Reinforcement learning from human feedback

Reinforcement learning from human feedback converts judgments about model outputs into a training signal. A model first learns from demonstrations written or selected by human annotators, usually through supervised fine-tuning. It then produces several answers to the same prompt, and annotators rank or compare them. Those comparisons train a preference model, which estimates which response people would favor. The language model is subsequently optimized to obtain high predicted reward while remaining close to its supervised starting point.¹

In many influential systems, the final optimization stage uses Proximal Policy Optimization (PPO), a policy-gradient method designed to limit excessively large updates. The resulting model is not simply memorizing preferred answers: it is adjusting the probabilities of generating responses across a broad prompt distribution.

RLHF became especially prominent as a method for aligning large language models with instructions and conversational expectations. The 2017 work by Paul Christiano and colleagues showed that human comparisons could guide reinforcement learning on tasks whose correct objective was difficult to specify directly.¹ Later language-model research applied the approach to helpfulness, truthfulness, formatting, refusal behavior, and dialogue quality.

The InstructGPT work described a three-stage pipeline—demonstration-based fine-tuning, preference-model training, and PPO-based optimization—and reported improved human ratings over a much larger unaligned model on its tested instruction-following tasks.² RLHF is also used in robotics, recommendation, game-playing, and other settings where experts can evaluate outcomes more readily than they can write a complete objective.

RLHF optimizes a proxy for human approval rather than human values themselves. Annotators may disagree, overlook factual errors, reward confident style, or apply inconsistent standards; the preference model can therefore encode noise, bias, and gaps in the collected data.⁴ A policy may also discover behaviors that exploit weaknesses in the reward model, a problem commonly called reward hacking.

Other difficulties include expensive annotation, uneven representation of cultures and expertise, susceptibility to sycophancy, and the possibility that optimization produces superficially compliant answers without robust understanding. Human feedback can improve practical behavior without proving that a model is truthful, safe in every context, or aligned with a single, universally agreed conception of human values. Evaluation outside the training distribution is therefore essential.

RLHF is a family of design choices rather than one fixed algorithm. Researchers vary the annotator instructions, sampling strategy, preference-model architecture, reward regularization, and policy optimizer; these choices can materially change the resulting behavior.⁴ Feedback may come from domain experts, ordinary users, or structured evaluations, and comparisons can be more reliable than asking people to assign absolute numerical scores.

Several approaches reduce or replace the explicit reinforcement-learning stage. Direct preference optimization trains a policy directly from preference pairs under a probabilistic objective, while Constitutional AI uses written principles and model-generated critiques in parts of the feedback process.³ These methods are related to RLHF but should not be treated as identical. The broader research problem is often called alignment, of which preference-based fine-tuning is only one component.

Preference model

A model trained to predict which of two or more candidate outputs human evaluators would prefer.

Reward model

A learned function that assigns a scalar training signal to an output; in RLHF it commonly represents predicted human preference.

Policy

The model being optimized, viewed as a system that assigns probabilities to actions such as generating tokens.

PPO

Proximal Policy Optimization, a policy-gradient algorithm that constrains update sizes to improve training stability.

Reward hacking

Behavior that scores well under a flawed or incomplete reward function while failing to achieve the intended objective.