Reinforcement learning with verifiable rewards (RLVR) is a way of training a language model by letting it attempt a task, checking the result with a program, and rewarding only answers that pass the check. It replaced the learned human-preference reward of earlier methods with a rule that can say “correct” or “incorrect”, and it became a core ingredient of today’s reasoning models. The term was introduced by the Allen Institute for AI’s Tülu 3 work, and the method gained wide attention through the DeepSeek-R1 paper. This explainer covers how it works, where it has been shown to help, and where it breaks.
From human feedback to checkable answers
In reinforcement learning from human feedback (RLHF), a separate reward model is trained on human preferences and then used to score the main model’s outputs. Our explainer on the InstructGPT paper describes that pipeline. It works for open-ended style and helpfulness, but the reward model is itself a learned approximation, and the policy can exploit its mistakes.
RLVR removes that middle step where the task allows. For a math problem with a known answer, a program compares the model’s final answer to the reference. For a coding task, unit tests run. For a formatting instruction (“answer in exactly three bullet points”), a script counts bullets. The reward is simple: Tülu 3’s authors describe giving a fixed reward when the verifier confirms the output and nothing otherwise. They applied it to math tasks and to precise instruction following, after supervised fine-tuning and preference tuning.
How the training loop works
A typical RLVR step looks like this:
- Sample a prompt with a verifiable answer, such as a math problem.
- Have the current model generate one or more complete responses, usually including a chain of reasoning.
- Run the verifier on each response and assign a reward.
- Update the model so that responses with higher rewards become more likely.
The update algorithm varies. The DeepSeekMath paper introduced Group Relative Policy Optimization (GRPO), which samples a group of responses to the same prompt and scores each one relative to the group average. This removes the need for a separate value network, which cuts memory and complexity. GRPO is used in the DeepSeek-R1 training described below, and it has become a common baseline.
What DeepSeek-R1 showed
The DeepSeek-R1 paper trained a variant called R1-Zero by applying large-scale reinforcement learning directly to a base model, without a supervised fine-tuning stage first. Rewards were rule-based: one for answer accuracy, one for following the required output format. According to the paper, the model’s pass@1 score on the AIME 2024 math contest rose from 15.6% to 71.0% over training. The authors also report that the model’s reasoning grew longer and began to include self-checking behaviors, which they present as emerging from the reward alone. The final R1 model added a small amount of cold-start data and further stages to improve readability and general ability.
Treat these as the authors’ own reported results on specific benchmarks; independent replications have varied in details, and benchmark scores depend on settings.
Where RLVR works and where it does not
The method needs a verifier, which determines its reach:
- Works well: mathematics with exact answers, code with tests, logic puzzles, structured-output tasks, and formal proofs checked by a proof assistant.
- Hard: writing quality, advice, summaries, and any task where “correct” is a matter of judgment. A learned or model-based judge can stand in, but then the reward is no longer fully verifiable.
- Risky: weak verifiers. If a test suite is incomplete, the model can learn to pass the tests without solving the problem, the behavior our article on reward hacking examines. A verifier is only as good as its coverage.
There is also a debate over what RLVR adds. One view is that it mainly makes the model more reliable at reaching answers it could already find occasionally; another is that long training can unlock new reasoning strategies. The cited papers do not settle that question, and we do not claim an answer.
A worked example
Suppose the prompt asks for the sum of the first 20 positive integers. The model produces a short reasoning chain and ends with “Answer: 210”. The verifier extracts the number after “Answer:”, compares it to 210, and returns a reward of 1. A response ending in “Answer: 200” receives 0. In GRPO, if the model sampled eight responses and five were correct, the correct ones get a positive advantage relative to the group average and the incorrect ones a negative one, so the update shifts probability toward the reasoning paths that reached 210.
What to watch next
- Verifiers for domains beyond math and code, such as science problems with computable answers.
- Methods that detect and penalize test gaming.
- Open replications and ablations of GRPO against other policy-gradient methods.
- Studies on whether RLVR expands what models can solve or sharpens what they already can.
Key takeaways
- RLVR rewards a model when a program confirms its answer is correct, instead of using a learned preference model.
- Tülu 3 named the approach; DeepSeek-R1 popularized it with rule-based rewards and GRPO.
- It is strongest where answers can be checked automatically and weakest on subjective tasks.
- Weak verifiers invite reward hacking.
Frequently asked questions
What does RLVR stand for?
Reinforcement learning with verifiable rewards. The reward comes from an automatic check of correctness, not from a reward model trained on human preferences.
What is GRPO?
Group Relative Policy Optimization, introduced in the DeepSeekMath paper. It scores each of several sampled answers against the group average, avoiding a separate value network.
Can RLVR be used for creative writing?
Not directly, because there is no program that can verify quality. Teams use preference models or model judges for such tasks, which are less reliable rewards.
