Every model launch comes with a chart. Bars are taller on the new model, the axis starts conveniently high and a footnote says “internal evaluation.” You do not need to be a researcher to read these charts well; you need a short list of questions and the habit of asking them in the same order. This cheat sheet is that list. It is built for quick use: skim the table, run the checks, and decide whether a claim is strong evidence, a hint or marketing. For background on why scores saturate and mislead, see our explainer on how AI benchmarks work.
The two-minute checklist
| Question | What a good answer looks like | Red flag |
|---|---|---|
| Who ran the test? | An independent party or a public leaderboard | Only the vendor, “internal” benchmark, no method |
| How many runs? | Several runs with a spread or confidence interval | A single run reported as one number |
| How many tasks? | Hundreds of tasks, size stated | Tiny test set, size not given |
| How big is the gap? | Larger than the rerun noise | A one- or two-point lead called a “win” |
| What setup was used? | Model, prompt, tools, settings and effort level named | “Best configuration” with no details |
| Was the test data seen in training? | Held-out or newly written tasks, contamination checks | Public benchmark with leaked answers, no mention |
| What did it cost? | Tokens or dollars per task reported | Only accuracy, no cost |
| Does it match my use? | Task type similar to your work | A benchmark from a different domain |
Check 1: noise first, then gaps
Scores move even when nothing changes. A 2026 preprint that reran the same coding-agent setups found that identical configurations can differ by up to six points on SWE-bench Verified, and that swapping the agent harness flipped about as many individual tasks as simply rerunning did. The authors, Yangze Liu and Zhongyi Han, argue that many advertised gains of two or three points sit inside that range (we summarize the preprint in our research note; it has not been peer reviewed). The older reinforcement-learning paper “Deep Reinforcement Learning that Matters” made a similar point about random seeds. Rule of thumb: if a claimed lead is smaller than the spread between reruns, treat it as a tie.
Check 2: sample size sets the smallest detectable difference
The fewer tasks in a test, the larger a gap has to be before you can trust it. The same preprint estimates that with 45 tasks even a 13-point gap shows up only about half the time, while with 447 tasks gaps near five points become reliably detectable. When a chart does not state the number of tasks, assume the worst. A benchmark with a few dozen items can swing by a lot from a handful of lucky or unlucky answers.
Check 3: the setup must be reproducible
Scores depend on prompts, tools, temperature, retries and how much “thinking” the model may use. A claim that says “with maximum reasoning effort and tools” is not comparable with another that used default settings. Look for a named configuration and compare like with like. The paper “Show Your Work” argues that results should report the compute and tuning budget used, because a method that gets many attempts will look better than one that gets one. If a vendor compares its best setting with a rival’s default, discount the gap.
Check 4: contamination and shortcuts
If a model has seen the test questions or their answers during training, the score measures memory, not skill. Public benchmarks are especially exposed, and agent tests can have shortcuts: the preprint above reports that, with web access enabled, a large share of some models’ attempts fetched the correct fix straight from GitHub, so the authors ran their final tests offline. Prefer benchmarks with held-out sets, regularly refreshed tasks or contamination checks, and treat perfect or near-perfect scores on old public tests with suspicion. Our piece on reward hacking explains why optimizing for a score can drift from the goal.
Check 5: look at cost, speed and breadth
Accuracy alone hides trade-offs. The preprint found that heavier agent harnesses used two to three times as many tokens as the leanest option without a measurable gain in pass rate. A model that is one point better but three times more expensive may be worse for your budget. Stanford’s HELM project argues for evaluating many scenarios and metrics, including robustness, calibration and efficiency, rather than a single headline number. When you can, look for results across several benchmarks and for latency and price per task.
Check 6: does it match your task?
A benchmark measures a specific skill: fixing Python issues, answering graduate-level science questions, using a computer. Your task may differ in language, domain, document length or tolerance for errors. Use scores to shortlist candidates, then run 20 to 50 of your own real examples through each. As our quick take on benchmark scores says, a leaderboard rank is a starting point for a test, not a purchase decision.
Quick reference: phrases and what to ask
- “State-of-the-art on…”: by how much, on how many tasks and against which setup?
- “Internal benchmark”: can anyone else reproduce it?
- “Matches human experts”: which experts, how many, and on which tasks?
- “Up to X% better”: under which best-case setting?
- “Pass@k”: how many attempts were allowed, and can a user get them in practice?
What to watch next
- Leaderboards that publish run-to-run spread and cost beside the score.
- Vendors reporting results over repeated runs by default.
- New benchmarks built from fresh, unpublished tasks to limit contamination.
- Independent evaluators gaining deeper access to frontier systems.
Key takeaways
- Check who ran the test, how many runs and how many tasks before reading the numbers.
- Differences smaller than rerun noise are ties, not wins.
- Compare setups like for like, and be wary of “best configuration” claims.
- Report cost and speed with accuracy; a small gain may not justify triple the tokens.
- Always validate on your own examples before choosing a model.
Frequently asked questions
What is a good minimum number of tasks for a benchmark?
There is no single number, but the cited preprint suggests that hundreds of tasks are needed to detect gaps of about five points; a few dozen tasks only reveal very large differences.
Are vendor-reported benchmarks useless?
No. They show what a vendor believes its model can do at its best. Treat them as a hint and look for independent results before relying on them.
How can I test a model for my own work?
Collect 20 to 50 real examples, run each model with the same prompt and settings, score the outputs against criteria you define, and repeat a few times to see the noise.
