A group of six researchers posted a preprint on October 23, 2026 showing that a language model can get better at standard benchmarks using nothing but a list of broad topic names and its own output. The method, called On-Policy Self-Training (OPST), lifted the average score of a small Gemma 4 model from 38.3 to 45.6 across three benchmarks, without human-written questions or answers. The paper has not been peer reviewed: the arXiv page lists no venue, so it should be read as a preprint.
The problem: improvement usually needs outside supervision
Most ways of improving a model after pretraining lean on something external. Supervised fine-tuning needs human-written answers. Reinforcement learning needs a reward, often a verifiable answer key. Even recent “self-distillation” methods, in which a model learns from a better-informed version of itself, usually need a curated set of problems or reference solutions.
The authors ask how far you can go with less. Their starting point is a coarse topic label, such as “Abstract Algebra”. From that single phrase the model has to create the questions, the reasoning and the answers. A related setup is covered in our explainer on knowledge distillation, where a student learns from a stronger teacher. Here, the teacher and the student are the same model.
How OPST works
The key idea is “privileged information”. For each topic, the model writes a question and, alongside it, a rationale and a candidate answer. The student version of the model sees only the question. The teacher version sees the question plus the rationale and answer. Training pushes the student’s token probabilities toward the teacher’s, measured on responses the student itself produced, which is what “on-policy” means.
Because the candidate answers come from the model, they can be wrong. The paper adds two filters:
- Pseudo-verification. A sample is kept only if the model reproduces the candidate answer when it solves the question again in a fresh, low-temperature context, and also when it solves a paraphrase of the question.
- Consensus gate. If every one of the student’s sampled attempts already agrees with the candidate answer, the sample is discarded as too easy to teach anything.
Two further design choices matter. The loss is a skewed Jensen-Shannon divergence (alpha of 0.8), a bounded measure chosen so that a few tokens with near-zero probability, which can appear when hints are imperfect, do not dominate the gradients. And the teacher is an exponential moving average of the student (rate 0.05), which is also the model that gets evaluated. The cycle of generating, verifying and training repeats over several rounds; seven rounds gave the best result in the authors’ tests.
A visual variant uses unlabeled OpenImages V4 pictures and topics such as spatial relations or object counting. Its privileged information is a generated scene caption or relations derived from detected bounding boxes.
What the numbers show
The main experiments use Gemma 4 E2B and E4B. Accuracy in percent, with the base model first:
- E2B, thinking off: MMLU 37.5 to 47.7, GPQA 28.8 to 32.8, HellaSwag 48.7 to 56.2; average 38.3 to 45.6. Plain supervised fine-tuning on the same data reached an average of 40.0 and GRPO reached 39.3.
- E2B, thinking on: average 58.4 to 63.0, with GSM8K moving from 72.7 to 80.9.
- E4B, thinking on: average 71.3 to 76.0, with GSM8K from 84.4 to 93.0 and MMLU from 76.0 to 80.1.
Compared with methods that get more help, OPST held up well. On E2B without thinking, a variant given curated questions averaged 44.6 and a version using human reference solutions averaged 46.6, against 45.6 for OPST. With thinking on, OPST’s 63.0 edged past the human-reference variant’s 62.8. In an appendix, Qwen3-4B-Base rose from an average of 66.7 to 72.7.
On images, the E2B model’s VQAv2 accuracy went from 49.1 to 56.0 and POPE F1 from 82.2 to 84.6, using only unlabeled photos.
Which parts matter
The ablations are the most informative section. On E2B MMLU, skipping both filters still gave 42.7, a 5.2-point gain over the base. Adding verification raised it to 45.4, and adding the consensus gate reached 47.7. Giving the teacher no extra information scored 40.6; giving it a placebo, a different problem on the same topic, scored 40.2; giving it the answer alone scored 44.6 and the rationale alone 45.8. The full package of question, rationale and answer was best. In other words, the gain comes largely from conditioning on the generated reasoning, not from copying it.
The method also tolerated bad data better than plain fine-tuning. When 75 percent of the candidate answers were deliberately wrong and filters were off, OPST fell 1.9 points to 40.8, while supervised fine-tuning fell 5.7 points to 33.1, below the untouched base model. Results across three random seeds varied by at most 1.2 points.
Limits the authors acknowledge
The authors state that OPST can only amplify knowledge a model already holds. Pseudo-verification measures self-consistency, not truth, so an error the model makes reliably can still slip into training. Topics are chosen by hand, and the tests cover small models only. Training on several domains at once also had costs: in one mixture, MATH-500 dropped from 56.5 to 51.6 even as the out-of-domain average rose from 43.6 to 44.4.
Our reading adds two cautions. All figures come from the authors’ own evaluations of a single preprint, and we have not seen independent replication. And the paper evaluates on a handful of standard benchmarks, so it says little about open-ended, real-world tasks. For context on how these stages fit together, see our overview of the post-training stack, and for how data made by models is used more broadly, the glossary entry on synthetic data.
What to watch next
- Replication on larger models, since the paper tests only small ones.
- Whether the filters can be improved to catch consistent but wrong answers.
- Independent checks on benchmarks the authors did not use.
- Whether the work appears at a peer-reviewed venue with reviewer feedback.
Key takeaways
- OPST trains a model using only topic names, with questions, rationales and answers all generated by the model itself.
- On Gemma 4 E2B without thinking, the three-benchmark average rose from 38.3 to 45.6.
- Plain fine-tuning and GRPO on the same data gained far less (40.0 and 39.3).
- The approach cannot add knowledge the model lacks, and filters check consistency, not correctness.
- The result is an unreviewed preprint with small models only.
Frequently asked questions
What is on-policy self-training?
It is a training method in which a model learns from a better-informed copy of itself, using its own generated questions and reasoning. The student answers without hints, and the teacher sees the generated rationale and answer.
Is the paper peer reviewed?
No. It is an arXiv preprint submitted on October 23, 2026, and the abstract page lists no journal or conference.
Does this mean models no longer need human data?
Not yet. The paper shows gains on small models and standard benchmarks, but the model still relies on knowledge learned from human data during pretraining, and the authors say the method only amplifies what is already there.
