PPO explained: the reinforcement learning algorithm behind RLHF

Some algorithms are famous for being clever. Proximal Policy Optimization (PPO) is famous for being reliable. Introduced by John Schulman and colleagues at OpenAI in 2017, PPO became the default reinforcement learning algorithm for a wide range of problems — from simulated robots to video games — and, most visibly, for the reinforcement learning from human feedback (RLHF) stage that turned language models into assistants.

Policy gradients and their problem

There are two broad families of reinforcement learning methods. Value-based methods, like DQN, learn how good each action is. Policy-gradient methods learn the policy directly: a neural network that outputs probabilities for each action, adjusted to make high-reward actions more likely.

The basic recipe is intuitive: try actions, see which ones led to better-than-expected results, and increase their probability. But it is fragile. Each update is computed from a limited batch of noisy experience. A step that is too large can change the policy so much that performance collapses — and because the policy also determines what data is collected next, a bad update can be hard to recover from.

The predecessor: trust regions

In 2015, Schulman and colleagues proposed Trust Region Policy Optimization (TRPO), which constrains each update so the new policy stays close to the old one, measured by a statistical distance (the KL divergence). TRPO was stable but mathematically complicated, required second-order optimization tricks and did not fit easily with common neural network architectures.

PPO’s simplification: clipping

PPO keeps TRPO’s spirit — don’t move the policy too far in one step — with a much simpler mechanism.

For each action in the collected data, PPO computes the probability ratio: how much more (or less) likely the new policy makes that action compared with the old policy. It multiplies this ratio by the advantage, an estimate of how much better the action was than average.

Then comes the key trick: the ratio is clipped to a narrow range around 1, typically between 0.8 and 1.2. The objective takes the more pessimistic of the clipped and unclipped versions. In effect, once an update has already made a good action sufficiently more likely, there is no further reward for pushing it even more in that batch. This removes the incentive for destructively large updates.

Because the objective is a simple function, PPO can be optimized with standard tools like Adam, and it can safely reuse each batch of data for several training epochs, improving sample efficiency.

Results and adoption

The paper showed PPO matching or outperforming other methods on continuous-control benchmarks (simulated robotic locomotion) and Atari games, while being far simpler to implement than TRPO.

PPO became the workhorse for many high-profile projects. OpenAI used large-scale PPO to train OpenAI Five, which defeated world champions at the video game Dota 2 in 2019. And in the InstructGPT paper of 2022, PPO was the algorithm used to optimize language models against a learned reward model of human preferences — the core of RLHF.

PPO in language model training

In RLHF, the “policy” is the language model, the “action” is generating text, and the “reward” comes from a reward model trained on human rankings. PPO’s conservatism is valuable here, combined with a penalty that keeps the model close to its starting point so it does not drift into strange text that fools the reward model.

PPO is also demanding in this setting: it needs several large models in memory at once (the policy, a value model, the reward model and a reference model). That cost motivated simpler alternatives such as Direct Preference Optimization and other methods that drop the value model.

Why it matters

PPO illustrates a recurring lesson in AI: the most influential algorithm is often not the most theoretically elegant but the one that works robustly with reasonable defaults. Its central idea — make progress, but in small, safe steps — remains embedded in how many modern AI systems are trained.

Frequently asked questions

What is the “advantage” in PPO?

An estimate of how much better an action turned out than the agent expected on average in that situation. Positive advantage means the action should become more likely; negative advantage means less likely.

Why is PPO called “proximal”?

Because it keeps each new policy close to — proximal to — the previous one. Clipping the probability ratio limits how far a single update can move the policy.

Is PPO still used for language models?

It remains common, but alternatives such as Direct Preference Optimization and value-free reinforcement learning methods are increasingly popular because they are cheaper and simpler to run at scale.