Post-training is everything done to a language model after pretraining to make it follow instructions, prefer good answers and solve tasks reliably. The main methods are supervised fine-tuning (SFT), preference optimization such as DPO, KTO and ORPO, and reinforcement learning, either RLHF with PPO or GRPO with verifiable rewards. For most teams adapting an open model, the right answer is SFT first, then DPO if you have preference data, and GRPO only when answers can be checked automatically.
This guide explains what each method needs, what it costs and when it is worth the trouble. It builds on our practical overview of LoRA fine-tuning, since all of these methods can be combined with LoRA adapters to cut memory use.
The standard pipeline in one paragraph
The recipe most open labs publish follows three stages. Supervised fine-tuning teaches the format and skills from example conversations. Preference tuning then teaches the model which of two acceptable-looking answers people prefer. A reinforcement learning stage pushes performance on tasks with a measurable outcome, such as math or code. The Allen Institute’s Tülu 3 report is a clear public example: SFT, then DPO, then reinforcement learning with verifiable rewards (RLVR). OpenAI’s 2022 InstructGPT paper established the older version, with SFT followed by a learned reward model and PPO.
Supervised fine-tuning (SFT)
What it needs: prompt and response pairs that show exactly the behavior you want. A few thousand high-quality examples is a common starting point for a narrow task.
How it works: the model is trained to predict the reference response token by token, the same objective as pretraining but on curated conversations. In Hugging Face’s TRL library this is the SFTTrainer.
Good for: teaching a format (JSON output, a house style, a support persona), a domain vocabulary or a narrow skill. It is the cheapest method and the most predictable.
Limits: the model can only imitate what it sees. If your examples contain mistakes, it learns them. SFT does not teach the model which of two plausible answers is better. Our guide on preparing a fine-tuning dataset covers how to build the examples.
DPO: preference tuning without a reward model
What it needs: preference pairs, each with a prompt, a preferred (“chosen”) answer and a worse (“rejected”) answer. The pairs can come from human raters, from user feedback such as thumbs up and down, or from a stronger model acting as a judge.
How it works: Direct Preference Optimization, introduced by Rafailov and colleagues in 2023, showed that the RLHF objective can be optimized directly on the pairs with a simple classification-style loss. The model raises the probability of the chosen answer relative to the rejected one, while a frozen copy of the starting model, the reference model, keeps it from drifting too far. There is no separate reward model and no sampling loop. We explain the math in our DPO paper explainer.
Good for: tone, helpfulness, refusing less or more, choosing between styles. It is stable and runs on the same hardware as SFT, with extra memory for the reference model.
Limits: it learns only from the pairs it is given, so it cannot discover better answers on its own. Poorly built pairs, where the chosen answer is simply longer, teach the model to be verbose.
KTO and ORPO: two simpler variants
KTO (Kahneman-Tversky Optimization, Ethayarajh and colleagues, 2024) works with unpaired feedback: each example is just a response marked desirable or undesirable. That matches the data many products already collect, where users rate single answers instead of comparing two.
ORPO (Odds Ratio Preference Optimization, Hong and colleagues, 2024) merges SFT and preference tuning into one stage and removes the reference model. It saves a training run and memory, at the cost of being less widely tested than DPO.
TRL ships trainers for both, so switching between DPO, KTO and ORPO is mostly a matter of reshaping the dataset.
RLHF with PPO
What it needs: preference data to train a reward model, then a large set of prompts for the reinforcement learning stage.
How it works: a reward model learns to score answers the way human raters would. The language model then generates answers, the reward model scores them and the PPO algorithm updates the model toward higher scores, with a penalty for moving too far from the starting point. In the InstructGPT paper, labelers preferred outputs from a 1.3-billion-parameter model tuned this way over those of the 175-billion-parameter GPT-3.
Good for: the model can explore and find answers better than any in the dataset, as long as the reward model judges them correctly.
Limits: it is the most complex and expensive option. PPO keeps several models in memory at once (the policy, a reference copy, the reward model and a value model), and training is sensitive to settings. Models also learn to exploit flaws in the reward model, a failure known as reward hacking.
GRPO and verifiable rewards
What it needs: prompts whose answers can be checked automatically, such as math problems with a known result, code with unit tests or outputs that must match a schema.
How it works: Group Relative Policy Optimization was introduced in the DeepSeekMath paper in 2024. For each prompt the model generates a group of answers, each is scored, and each answer’s advantage is measured against the group average. This removes PPO’s separate value model, which cuts memory. When the score comes from a checker instead of a learned reward model, the setup is often called reinforcement learning with verifiable rewards.
Good for: reasoning, math, coding and any task with a reliable automatic grader. It is the method behind much of the recent progress in open reasoning models.
Limits: it needs a good checker, and it needs a base model that already gets some answers right, since a group in which every answer scores zero gives no learning signal. Generating many samples per prompt also makes it slower than DPO.
A worked example: a support assistant
Suppose a software company wants an open model to answer support tickets in its own style. It has 5,000 past tickets with good agent replies, a log of replies customers rated, and a test suite that checks whether suggested configuration snippets are valid.
- SFT on the 5,000 ticket and reply pairs teaches the format, product names and tone.
- KTO on the rated replies, or DPO if the team builds pairs from them, shifts the model toward answers customers found useful.
- GRPO on the configuration questions, using the test suite as the reward, improves the one part of the job where correctness can be checked.
Most teams can stop after step 2. Step 3 pays off only if the checkable part matters and the checker is trustworthy. Whatever the path, compare every stage with the previous one on a held-out set, as described in our guide to testing a fine-tuned model before you ship it.
Quick comparison
| Method | Data needed | Extra models in memory | Relative cost | Best for |
|---|---|---|---|---|
| SFT | Example answers | None | Low | Format, style, narrow skills |
| DPO | Chosen and rejected pairs | Reference model | Low to medium | Tone and preference alignment |
| KTO | Single answers rated good or bad | Reference model | Low to medium | Product feedback logs |
| ORPO | Chosen and rejected pairs | None | Low | One-stage SFT plus preferences |
| RLHF with PPO | Pairs, then prompts | Reward, value and reference models | High | Open-ended quality with a strong reward model |
| GRPO | Prompts plus a checker | Reference model, optional reward model | Medium to high | Math, code, verifiable tasks |
What to watch next
- Open post-training recipes from labs such as Ai2 that publish full data and code.
- Cheaper reinforcement learning methods that reduce the number of samples needed per prompt.
- Better automatic checkers for tasks that are not math or code.
- Hardware and services aimed at private fine-tuning and reinforcement learning on sensitive data.
Key takeaways
- Start with SFT; it is cheap, stable and enough for format and style.
- Use DPO, KTO or ORPO when you have preference or rating data and want better judgment between answers.
- RLHF with PPO is powerful but complex, costly and prone to reward hacking.
- GRPO works best when answers can be verified automatically, such as math or tested code.
- Evaluate after every stage on held-out data; each method can improve one skill while hurting another.
Frequently asked questions
What is the difference between DPO and RLHF?
RLHF trains a separate reward model and then uses reinforcement learning, usually PPO, to optimize against it. DPO skips the reward model and the sampling loop and trains directly on pairs of preferred and rejected answers.
Is GRPO better than PPO?
GRPO uses less memory because it drops PPO’s value model and compares each answer with others in its group. It works best with verifiable rewards; PPO remains common when rewards come from a learned model.
Do I need reinforcement learning to fine-tune a model?
No. Many practical projects only need supervised fine-tuning, sometimes followed by DPO. Reinforcement learning is worth it when you have a reliable way to score answers and need gains beyond your examples.
