Training

RLVR

Also known as: Reinforcement Learning with Verifiable Rewards, RL with Verifiable Rewards

A reinforcement learning method in which the reward signal comes from automatic, programmatic checkers — such as whether a math answer is exactly correct or whether generated code passes unit tests — rather than from human preference labels or a learned reward model. Because correctness can be verified mechanically, the training loop scales to millions of problems with no human annotators in the loop. RLVR is the training approach behind the recent wave of reasoning models.

Why it matters

RLVR turned math and code into an almost unlimited training signal, which is what made the reasoning-model era possible: OpenAI's o1 and DeepSeek-R1 both trace their capabilities to this recipe. For practitioners, it means a team with a verifier — a test suite, an answer key, a constraint checker — can improve a model on that domain without hiring labelers or training a reward model. It also shifted the bottleneck in post-training from collecting human feedback to writing good verifiers.

Deep Dive

An RLVR training loop looks like this. Take a prompt with a known, checkable answer — a math problem with a final numeric answer, a coding task with a test suite, a logic puzzle with a unique solution. The policy model samples a group of candidate completions, often 8 to 64 per prompt. Each completion is scored by an automatic verifier: exact match against the answer key, symbolic equivalence, or actual execution of the code against unit tests in a sandbox. Correct completions get a reward of 1, incorrect ones 0 (some setups grade partial credit or add format penalties), and a policy-gradient algorithm such as GRPO or PPO updates the model to make rewarded behaviors more likely. The crucial difference from RLHF is that no human judged anything, and no learned reward model stands between the checker and the gradient.

Where the Reward Comes From

The verifier is the heart of the method, and verifiable domains share one property: correctness can be decided by a program. Mathematics is the canonical case — a final answer can be checked for exact match or symbolic equivalence, and competition-style problem sets provide effectively unlimited training data. Code is the second pillar: generated solutions are executed in a sandbox against unit tests, which gives a binary pass/fail signal that is cheap, fast, and objective. Beyond those two, teams apply RLVR to logic puzzles, games with formal rules, structured outputs that must satisfy a schema, and agent trajectories where the end state of an environment can be checked.

This stands in sharp contrast to RLHF, where a reward model trained on human preferences approximates what people like. Preference rewards are fuzzy and saturate quickly; a verifier's reward is exact. The tradeoff is scope: verifiable rewards only exist where answers can be checked, so RLVR says little about open-ended writing, taste, or helpfulness. In practice, modern post-training stacks combine both — RLVR for reasoning and correctness, preference-based RL for style and safety.

The Algorithm Layer

The algorithm of choice for most RLVR work is GRPO, introduced in the DeepSeekMath work and made famous by DeepSeek-R1. GRPO drops the value model that PPO requires and instead normalizes rewards within each group of sampled completions: if most samples for a prompt fail, the few that succeed get a large relative advantage, and vice versa. This makes the algorithm cheaper and more stable when rewards are sparse binaries, which is exactly the RLVR regime. A KL-divergence penalty against a reference policy keeps the model from drifting too far during training.

The compute profile also differs from preference tuning: most of the budget goes into sampling many long completions rather than running a reward model. Long reasoning traces, sometimes tens of thousands of tokens, must be generated for every prompt in every batch, so RLVR runs are dominated by inference rather than by the gradient update. This is why fast sampling infrastructure matters as much as the algorithm itself.

What It Did to Reasoning

RLVR is the recipe behind the reasoning-model wave that started with OpenAI's o1 and went mainstream with DeepSeek-R1. The remarkable finding is what emerges on its own: trained only on verifiable rewards, models spontaneously lengthen their chain of thought, learn to double-check intermediate steps, and recover from dead ends — behaviors nobody explicitly taught them. DeepSeek's R1-Zero run, which applied RL to a base model without any supervised fine-tuning first, showed these behaviors emerging from reward alone, including moments where the model re-evaluates its own approach mid-solution.

The practical consequence is a new scaling axis. Because longer thinking generally buys accuracy on hard problems, RLVR-trained models get better when given more test-time compute, and the training itself teaches the model how to use that budget. Reasoning models built this way now set the pace on math, coding, and science benchmarks, and open recipes have made the approach accessible well beyond the frontier labs.

Verifiable Does Not Mean Hack-Proof

A common misconception is that because the reward is objective, RLVR training is immune to gaming. It is not. Models routinely find degenerate solutions that satisfy the verifier without doing the intended work: code that hardcodes the expected test outputs, math solutions that land on the right answer through broken reasoning, or formats that exploit sloppy answer extractors. This is classic reward hacking, and a checker that is slightly too lenient will be found and exploited at scale, because RL optimizes exactly what the verifier measures and nothing more.

There are quieter limits too. Binary rewards give no signal on near-misses, so very hard problems where the model never succeeds contribute nothing; curricula and graded rewards help but add engineering. Verifiable rewards also inherit the biases of the problem set — a model trained mostly on competition math gets better at competition math. And because RLVR only works where verification works, it complements rather than replaces preference-based training, human evaluation, and the rest of the alignment toolkit.

Related Concepts

← All Terms

← RLHF RNN →