GRPO
Why it matters
Deep Dive
GRPO — Group Relative Policy Optimization — first appeared in DeepSeek's DeepSeekMath paper in 2024, then became famous a year later as the training algorithm behind DeepSeek-R1, the open model that showed reinforcement learning alone could elicit long, self-checking chains of thought. At its core it is a pragmatic simplification of PPO, the algorithm that powered classic RLHF pipelines. PPO trains a second large model, the critic, to estimate how good a partially generated answer is; GRPO throws the critic away. For each prompt, the current policy samples a group of candidate outputs, each output gets a reward, and each one's advantage is simply how far its reward sits above or below the group average. The result is an RL loop that is cheaper to run, easier to tune, and a natural fit for the verifiable rewards used to train reasoning models.
How the Group Replaces the Critic
In PPO, the critic's job is to predict the expected reward of a partial answer so the algorithm can tell whether the final outcome was better or worse than expected. GRPO gets the same baseline signal from statistics instead of a neural network. Given a prompt, the policy generates a group of G outputs — anywhere from 8 to 64 in practice — and each output is scored, either by a rule-based checker (did the math answer match, did the code pass the tests) or by a learned reward model. The advantage of each output is then its reward minus the group mean, divided by the group standard deviation. Outputs above the group average are pushed up in probability, outputs below it are pushed down, and a PPO-style clipped objective plus a KL penalty toward a frozen reference model keeps each update small. Because the baseline is computed per prompt, prompts where every sample scores the same contribute almost no gradient, which turns out to matter a lot in practice.
Why It Is Cheaper and More Stable
The savings are concrete. Classic PPO pipelines hold four models in memory during training — the policy, a frozen reference policy, a reward model, and a critic that is usually as large as the policy itself — and the critic has to be trained alongside everything else, which is notoriously finicky. GRPO removes the most troublesome of the four, freeing VRAM for bigger batches or larger models and eliminating a whole class of value-learning bugs. Group normalization also handles reward scale automatically: a prompt whose rewards are tightly clustered gets its advantages squashed toward zero, so the update focuses on prompts where the model's outputs genuinely differ in quality. The tradeoff is that the compute budget shifts from training the critic to inference: generating 16 or 32 samples per prompt across thousands of prompts per step means the sampler, not the optimizer, is usually the bottleneck.
It Is an Optimizer, Not a Reasoning Recipe
A common misconception is that GRPO is inherently tied to reasoning or verifiable rewards — that the algorithm itself produced the behavior seen in DeepSeek-R1. It does not. GRPO is reward-agnostic: it will happily maximize whatever signal it is given, including a learned reward model scoring open-ended qualities like helpfulness or style. The reasoning breakthrough came from the combination — GRPO plus rule-based verifiable rewards, plus a strong base model, plus training runs long enough for behaviors like self-verification to emerge — the recipe now described as RLVR. Conversely, verifiable rewards work fine with plain PPO, and several labs trained reasoning models that way before GRPO existed. Keeping the two ideas separate matters practically: teams that adopt GRPO expecting reasoning to emerge from the algorithm alone, without carefully designed rewards and enough compute, are usually disappointed.
Where It Breaks Down
GRPO's elegance comes with failure modes worth knowing. The most discussed is the all-or-nothing group: when every sample for a prompt gets the same reward — all correct or all wrong — every advantage is zero and the prompt contributes nothing, so the training signal depends heavily on keeping prompt difficulty in the band where the model succeeds sometimes. Reward sparsity makes this worse on hard tasks, which is why practical pipelines mix in easier problems or use partial credit. Small groups give noisy baselines; large groups cost more sampling compute, and the right size is still a matter of empirical tuning. Over long runs, models can also suffer entropy collapse, where the policy becomes overconfident too early and stops exploring, or length blow-up, where answers grow longer because length happens to correlate with reward. Most of these issues have mitigations, and implementations in frameworks like Hugging Face TRL expose the relevant knobs, but GRPO is not a fire-and-forget algorithm.