Skip to main content
Zubnet AI › Learn › Wiki › Process Reward Model
Training

Process Reward Model

Also known as: PRM, Step-Level Verifier
A model that scores each intermediate step of a reasoning chain, rather than only the final answer, so that good reasoning can be guided and rewarded as it unfolds.

Why it matters

Step-level feedback fixes two weaknesses of outcome-only rewards. First, it's far more informative: one signal per step instead of one per problem. Second, it rewards being right for the right reasons. An outcome model can reward a solution that reaches the correct answer through flawed logic, while a PRM flags the flaw.

PRMs are also a key tool for test-time compute. If you can score partial solutions, you can search: explore several lines of reasoning, drop the weak ones early and spend effort on the promising ones.

Deep Dive

There are two ways to grade a student's math solution. You can check only the final answer, or you can read the working line by line and mark the first step that goes wrong. An outcome reward model (ORM) does the first: it sees a whole solution and predicts whether the result is correct. A process reward model (PRM) does the second: it gives every step its own score.

The idea was popularized by OpenAI's 2023 paper "Let's Verify Step by Step", which trained a PRM on about 800,000 human labels of individual solution steps and showed it picked correct math solutions more reliably than an outcome-based grader.

Using a PRM at Inference Time

  • Best-of-N reranking: generate many complete solutions, score each one's steps and return the solution with the best overall score.
  • Step-level beam search: generate several candidate next steps, keep the top few according to the PRM and continue from those.
  • Tree search: use PRM scores to guide a search over possible reasoning paths, in the spirit of AlphaGo's value network.

Experiments have shown that a small model with a good PRM and enough search can match a much larger model answering in one pass on math benchmarks.

Using a PRM for Training

A PRM can supply dense rewards for reinforcement learning, or filter synthetic reasoning traces so only sound ones are used as training data.

Where the Labels Come From

Human step labels are expensive, since they require experts to read reasoning line by line. Automated alternatives estimate a step's quality by rollouts: continue the solution many times from that step and see how often the final answer turns out correct. Another route is to have a strong LLM act as the judge of each step.

Limits

  • What counts as a step? Math solutions break into steps neatly. General reasoning and agent tasks don't.
  • Judging a step is hard. A step can look fine in isolation and still lead nowhere.
  • Reward hacking. A policy trained against a learned PRM can learn to produce steps the PRM likes instead of steps that are right.
  • Domain narrowness. A PRM trained on math doesn't transfer well to code or science.

For these reasons, the best-known reasoning models have leaned mostly on outcome rewards that can be checked automatically (the answer matches, the tests pass) at very large scale. DeepSeek's R1 paper, for instance, explicitly discussed PRMs and chose not to rely on them. PRMs remain valuable where verification of the final result is expensive and where search at inference time pays off.

← All Terms
ESC