A freshly pretrained language model is not an assistant. It is a very good predictor of the next token, trained on a vast pile of text, and if you ask it a question it may continue with another question, a forum thread or a block of boilerplate. Everything that turns that predictor into something that follows instructions, refuses some requests, shows its working and writes usable code happens afterward, in a phase the field calls post-training. This deep dive walks through the stages in the order they usually run, explains what each one changes, and shows where each one breaks. It is a synthesis of published research, not a description of any one company’s private pipeline, which in most cases is not public.

Why post-training exists

Pretraining optimizes a single objective: predict the next token of web-scale text. That objective rewards modeling every kind of document, including bad ones. It says nothing about being helpful, honest or safe, and it does not teach the conversational format.

The clearest early evidence that the next stage matters came from OpenAI’s InstructGPT work. In the paper, the authors report that human labelers preferred outputs from a 1.3 billion parameter model fine-tuned with human feedback over outputs from the 175 billion parameter GPT-3, despite the smaller model having more than one hundred times fewer parameters. The lesson was that behavior is shaped by the training signal, not only by scale. Our explainer on the InstructGPT paper covers that result in more detail.

Post-training is usually cheaper than pretraining by orders of magnitude in compute, but it is not cheap in people, data and iteration. Data quality dominates, and the evaluation burden is heavy. That is why open recipes such as Ai2’s Tulu 3 report, which publishes data, code and training recipes together, have been valuable to the community.

Stage one: supervised fine-tuning

The first stage is supervised fine-tuning (SFT), sometimes called instruction tuning. The model is trained on pairs of prompts and good responses, written by people or curated from other sources, using the same next-token loss as pretraining but only on the response tokens. In the InstructGPT recipe this is the “demonstration” step: labelers write ideal answers to prompts and the model learns to imitate them.

SFT does three jobs at once:

  • It teaches the chat format, including roles, turn-taking and stopping.
  • It unlocks abilities the base model already has but does not reliably show, such as following a multi-part instruction.
  • It sets a floor of style and safety behavior.

SFT also has a ceiling. The model can only imitate what is in the data, and it is graded token by token against one reference answer, even when many different answers would be equally good. It cannot learn “this answer is better than that one” because it never sees the bad answer. For that, you need comparisons.

For practical advice on the data side, see our guide on preparing a dataset for fine-tuning.

Stage two: learning from preferences

Preference learning starts from a simple observation: people find it easier to rank two answers than to write a perfect one. In the classic reinforcement learning from human feedback (RLHF) pipeline described in the InstructGPT paper, labelers rank several model outputs for the same prompt. A separate reward model is trained to predict those rankings, and the language model is then optimized with a reinforcement learning algorithm, Proximal Policy Optimization (PPO), to produce outputs that score highly. A penalty keeps the tuned model from drifting too far from the SFT model.

The pipeline works, but it is heavy: it needs a reward model, a policy, a reference model and, in PPO’s standard form, a value model, all in memory and all tuned carefully.

Direct preference optimization

The Direct Preference Optimization (DPO) paper showed that you can skip the explicit reward model and the reinforcement learning loop. Its authors derive a loss that is applied directly to pairs of a preferred and a rejected response, and they present it as a simple classification-style objective that is stable and cheap to run. In effect the language model itself plays the role of the reward model, which is the joke in the paper’s title. DPO is a main reason preference tuning became accessible to small teams. We explain the method in our DPO paper explainer.

DPO is not a free lunch. It learns from a fixed dataset of comparisons collected from some earlier model, so the data can drift out of date as the policy improves. Online methods that sample fresh responses from the current model and score them address that, at the cost of more machinery.

Feedback from models, not only people

Human labels are slow and expensive. Anthropic’s Constitutional AI paper describes a way to reduce reliance on them for harmlessness. In a supervised phase, a model critiques and revises its own responses against a written list of principles. In a reinforcement learning phase, a model generates the preference labels (an approach often called reinforcement learning from AI feedback), which then train a preference model. The same idea of using a strong model as a judge underlies many open datasets today, with the caveat that the judge’s biases flow into the student.

Stage three: reinforcement learning with verifiable rewards

For tasks with checkable answers, such as math with a known result or code that must pass tests, you do not need a learned reward model at all. You can run a checker. This idea, reinforcement learning with verifiable rewards (RLVR), appears in the Tulu 3 report as one of its post-training stages after SFT and preference tuning, applied to domains such as math and precise instruction following.

Diagram of the reinforcement learning loop: an agent takes an action, the environment returns a new state and a reward
The basic reinforcement learning loop that preference and verifiable-reward training build on: act, receive a reward, update. Image: Megajuice / Wikimedia Commons, CC0

Two research results pushed this direction into the mainstream:

  1. GRPO. The DeepSeekMath paper introduced Group Relative Policy Optimization, a variant of PPO that drops the separate value network. For each prompt it samples a group of responses, scores them and uses the group’s average as the baseline, so each response is judged relative to its siblings. That cuts memory and complexity.
  2. Reasoning from reward alone. The DeepSeek-R1 paper reports applying large-scale reinforcement learning with rule-based rewards to produce strong reasoning behavior. Its R1-Zero variant was trained with reinforcement learning without supervised fine-tuning as a preliminary step, and the authors describe behaviors such as longer chains of thought and self-checking emerging during training. The final R1 model adds some supervised data and further stages to fix readability and language mixing.

The practical consequence is that “reasoning models” are, to a large degree, a post-training story: the same base model, trained with a different reward signal and given room to think before answering. Our glossary entry on reasoning models gives the short definition.

Where each stage fails

Every stage optimizes a proxy, and proxies get gamed.

  • SFT fails by imitation. It copies the style of its data, including confident tone without correctness, and can teach a model to refuse or comply in the patterns it saw.
  • Preference methods fail by taste. Reward models and judges tend to prefer longer, more fluent, more agreeable answers. Optimize hard against them and you get verbosity and flattery. The InstructGPT-style KL penalty and careful early stopping exist because of this.
  • Verifiable rewards fail by loopholes. If the checker has a gap, such as unit tests the model can special-case, the model will find it. Our piece on reward hacking walks through examples and mitigations.
  • All stages fail by coverage. Behavior outside the training distribution is not constrained. A model that is polite in English chat may behave differently in a tool-calling loop.

A further risk is regression. Gains from one stage can erode capabilities from another, which is why teams mix general data into later stages and track a broad suite of evaluations at each step, not just the target metric.

What this means for builders

Most teams will never run a full pipeline. But the stack explains choices you do face:

  • If you need a format or style, SFT on a few thousand clean examples is usually enough.
  • If you need the model to prefer one kind of answer over another, preference tuning on pairs is the next step, and DPO-style methods are the low-cost entry point.
  • If you have an automatic checker, verifiable rewards can improve a narrow skill, with a real risk of exploitation if the checker is weak.
  • In every case, build the evaluation set before you train. See our guide on building an evaluation set for LLM apps.

What to watch next

  • More open recipes that publish data and code together, following the Tulu 3 example, which make results reproducible.
  • Wider use of verifiable rewards beyond math and code, in areas where correctness can be checked automatically.
  • Research on reward hacking and on judges that are harder to game.
  • Online and on-policy preference methods narrowing the gap with full RLHF at lower cost.
  • Evaluation practices that test for regressions across stages, not only the headline benchmark.

Key takeaways

  • Post-training, not pretraining, is what turns a base model into an assistant, and the training signal matters as much as scale.
  • The usual order is supervised fine-tuning, then preference optimization, then reinforcement learning where answers can be verified.
  • DPO removes the reward model and the reinforcement learning loop by training directly on preference pairs.
  • GRPO and rule-based rewards helped make reasoning-focused training cheaper and more widely reproducible.
  • Each stage optimizes a proxy that can be gamed, so evaluation before and after every stage is essential.

Frequently asked questions

Is RLHF still used?

Yes, in various forms. The original reward-model-plus-PPO recipe is one option; many teams use DPO-style or online preference methods, and add verifiable-reward training where it applies. Exact current practice at frontier labs is mostly not published.

What is the difference between SFT and preference tuning?

SFT trains the model to imitate good answers. Preference tuning shows it pairs of better and worse answers so it can learn what to move toward and away from.

Can I do any of this on a single GPU?

Parameter-efficient methods such as LoRA make SFT and DPO feasible on modest hardware for small and mid-sized open models. Our LoRA practical overview explains the trade-offs.