PPO
Why it matters
Deep Dive
PPO is a policy-gradient method, meaning it improves a policy — in language-model work, the model itself — by nudging it toward actions that scored well and away from actions that scored poorly. The signature trick is the clipped surrogate objective: for each action, PPO computes the ratio between the probability the current policy assigns it and the probability the previous policy assigned, multiplies that ratio by an advantage estimate of how much better the action was than expected, and then clips the ratio to a narrow band (commonly 0.8 to 1.2) so no single batch can yank the policy too far. Because clipping makes aggressive updates safe, the algorithm can run several epochs of minibatch gradient updates over the same collected data, which is far more sample-efficient than naive policy gradients. That combination — stability plus decent sample efficiency — made PPO the default reinforcement learning workhorse for years, first in games and robotics benchmarks and then in language model alignment.
The Machinery Around the Objective
PPO rarely runs alone. It is an actor-critic method, so alongside the policy (the actor) it trains a value model (the critic) that predicts the expected reward of a state, which lets it compute advantages through generalized advantage estimation rather than waiting for final scores. In the RLHF setting the memory bill grows further: a frozen copy of the original model serves as a reference policy, and a separate reward model supplies the scores, so a full training run can hold four models in memory at once. The reference model exists because the objective includes a penalty on the KL divergence between the current policy and the reference, which keeps the model from drifting into degenerate text that happens to score well. Getting this ensemble to train smoothly means juggling several interacting hyperparameters — the clip range, the KL penalty coefficient, the learning rate, and the advantage-estimation settings — which is a large part of PPO's reputation for being fiddly.
How PPO Powers RLHF
The pipeline that made PPO famous starts with a model that has already been through pre-training and supervised fine-tuning on demonstrations. Human annotators then rank several sampled responses to the same prompt, and those rankings are distilled into a reward model that predicts which responses people will prefer. PPO takes over from there: the model generates responses to fresh prompts, the reward model scores them, and the clipped objective plus the KL penalty updates the weights to favor higher-scoring behavior. This is essentially the recipe behind OpenAI's InstructGPT and ChatGPT, and it turned RLHF from a research idea into the standard way to make a base model helpful and follow instructions. The catch is reward hacking — push the optimization too hard and the model learns to exploit quirks of the reward model rather than produce genuinely better answers, which is why the KL anchor and careful early stopping matter so much in practice.
It Is Not Synonymous with RLHF
A common shorthand treats PPO and RLHF as the same thing, but they live at different levels: RLHF is the overall paradigm of training from human preferences, while PPO is just one optimizer you can plug into it. DPO skips reinforcement learning entirely, fitting the policy directly on preference pairs with a simple classification-style loss, and it works well for many alignment tasks. GRPO, popularized by DeepSeek's reasoning-model work, keeps the RL loop but throws out the value model, estimating each response's baseline from the average reward of a group of samples drawn for the same prompt. Removing the critic cuts memory and eliminates a whole category of tuning headaches, which is why GRPO has become a common default for training reasoning models on verifiable rewards. It is also worth remembering that PPO itself predates the LLM era — it was designed as a general-purpose algorithm for game-playing agents and simulated robot control, and only later got drafted into language model alignment.
The Practical Tradeoffs
Choosing PPO today is an engineering judgment call. On the plus side, it is battle-tested at the largest scales, its per-token credit assignment through the value model gives a fine-grained learning signal, and years of accumulated tooling and practitioner folklore make its failures diagnosable. On the minus side, holding four models in GPU memory is expensive, the hyperparameter surface is large and unforgiving, and reward over-optimization is a constant background risk: reward model scores can keep climbing while human judges say quality is actually falling. Teams with mature infrastructure and a need for tight control over training dynamics still tend to pick PPO, especially when rewards are noisy rather than binary. Teams that want something simpler, cheaper, and easier to reproduce increasingly start with GRPO or DPO and only reach for PPO when those methods hit a wall. Either way, understanding PPO remains the price of admission — nearly every modern post-training algorithm is best understood as a modification of it.