Post-Training
Why it matters
Deep Dive
A base model fresh out of pretraining is a powerful but awkward artifact: it continues text instead of answering questions, has no concept of being an assistant, and will imitate the worst of its training data as readily as the best. Post-training converts that artifact into a product through a sequence of stages, each with its own data, objective, and failure modes. The classic recipe — popularized by OpenAI's InstructGPT in 2022 and now used in some form by every major lab — starts with supervised fine-tuning on demonstrations, moves to preference optimization, and increasingly adds a heavy dose of reinforcement learning. The whole pipeline costs a small fraction of pretraining compute, yet it determines almost everything a user actually experiences: style, format, refusals, and how the model reasons.
Supervised Fine-Tuning Comes First
The first stage is instruction tuning: supervised fine-tuning on input-output pairs that show the model what a good assistant response looks like. The datasets are tiny by pretraining standards — thousands to millions of curated examples rather than trillions of tokens — and quality dominates quantity, because the model absorbs whatever habits the demonstrations contain. Much of this data is now synthetic, generated and filtered by stronger models instead of paid annotators. SFT also installs the chat template, the special tokens that mark system, user, and assistant turns, which is why the base and instruct versions of the same model behave so differently. Overdo this stage and the model starts losing capabilities from pretraining, a mild form of catastrophic forgetting that surfaces as regressions on skills nobody touched.
Preference Tuning with RLHF and DPO
Demonstrations cannot teach everything, so the next stage optimizes preferences: human raters rank pairs of model responses, the rankings train a reward model, and the policy is tuned to score well against it. The original recipe runs reinforcement learning, usually PPO with a KL penalty that keeps the policy from drifting too far, and this RLHF pipeline is what turned GPT-3-class base models into ChatGPT. DPO (2023) collapsed the whole loop into a supervised-style objective applied directly to preference pairs, no reward model or RL machinery required, which put preference tuning within reach of anyone who can run a fine-tune. Neither method is free: reward models get over-optimized as the policy learns to exploit their blind spots, and preference tuning is a known driver of sycophancy, since raters tend to reward agreeable, confident, flattering answers.
The Reasoning Era Changed the Math
The biggest recent shift is reinforcement learning with verifiable rewards: instead of human judgment, the reward comes from an automatic checker — does the math answer match the key, does the code pass the unit tests. Because it needs no labelers, this RLVR style of training scales far beyond preference data, and algorithms like GRPO, introduced by DeepSeek in 2024, cut the cost further by dropping the value model entirely. This is the recipe behind the reasoning-model generation — OpenAI's o1, DeepSeek-R1, and their successors — where models learn long chains of thought, backtracking, and self-correction through RL rather than imitation. A popular shortcut is distillation: train a smaller model on the long reasoning traces of a frontier model and capture much of the behavior without running RL at all, as DeepSeek did with R1's smaller Qwen and Llama variants. Either way, post-training compute, once an afterthought, is now a first-class scaling axis alongside test-time compute.
Safety Lives Here Too
Almost everything a user experiences as safety is post-training: refusal behavior, handling of dangerous requests, and robustness under jailbreak pressure. Labs mix safety examples into SFT, add harmlessness signals to the reward model, and hammer checkpoints with red-teaming before release; Anthropic's Constitutional AI replaces some human harmlessness labels with AI feedback checked against a written set of principles. Calibration is the hard part — too little safety training and the model is dangerous, too much and it refuses security research, medical questions, and dark fiction. These behaviors also sit near the surface: a few hundred steps of fine-tuning can strip them from an open-weights model, which is a live concern for every open release.
It Shapes Behavior More Than Knowledge
A persistent misconception is that post-training teaches the model new facts. Mostly it does not: the evidence, including the superficial alignment hypothesis behind work like LIMA in 2023, suggests that nearly all knowledge comes from pretraining, while post-training mainly surfaces, formats, and constrains what is already there. That has practical consequences. If a model lacks a capability, a bigger SFT dataset rarely fixes it — the knowledge has to come from the base model, or the RL stage needs a verifiable task where the behavior can actually be practiced. It also explains why counting on post-training to patch factual gaps is a risky plan, and why retrieval or a different base model is usually the honest answer.