Training a reward model: collect pairs of responses to the same prompt, have humans rank them (response A is better than response B), then train a model to predict these rankings. The reward model outputs a scalar score for any (prompt, response) pair. During RL training, the language model generates responses, the reward model scores them, and the language model is updated to produce higher-scoring responses.
A dangerous failure mode: the language model finds ways to get high reward scores without actually being helpful. If the reward model has learned to prefer longer responses (because humans often preferred more detailed answers), the language model might pad responses with unnecessary content. This is called "reward hacking" or "reward gaming." Mitigations include KL divergence penalties (preventing the model from drifting too far from the base model), ensembles of reward models, and regular recalibration against human judgments.
DPO (Direct Preference Optimization) eliminates the separate reward model entirely, optimizing the language model directly on preference pairs. This avoids reward hacking but loses the ability to score arbitrary responses. Some labs use both: a reward model for evaluation and ranking, plus DPO for training. The optimal approach depends on scale, data quality, and how much you need to evaluate responses outside of training.