Agent benchmark tables are full of compact labels such as pass@1, pass@4 and cost per rollout. They mean different things, and mixing them up leads to wrong comparisons. This cheat sheet defines each term, shows the formula where there is one and walks through a real example from the new CWE-bench v2 results. It extends our cheat sheet on reading benchmark claims.
The core vocabulary
Task. One problem with a checker, such as “fix this security weakness in this repository.”
Attempt, trial or rollout. One run of a model or agent on a task. Agents are not deterministic, so the same task can succeed on one run and fail on the next.
Reward. The checker’s score for a single attempt. For many coding benchmarks it is 1 for pass and 0 for fail, but some give partial credit.
Verifier. The program that decides the reward. Deterministic verifiers, which return the same result for the same output, make a benchmark reproducible.
pass@1: the average single-shot success rate
pass@1 is the probability that one attempt solves a task. If you run several attempts per task, you estimate it as the mean reward over all attempts. It answers the question: if a user runs the agent once, how often does it work? It is the headline number on most leaderboards.
pass@k: success within k tries
pass@k was popularized in the 2021 paper “Evaluating Large Language Models Trained on Code” by Chen and colleagues, which introduced the HumanEval benchmark. It is the probability that at least one of k sampled attempts solves the task. The paper’s unbiased estimator generates n attempts per task, counts c correct ones, and computes 1 minus the ratio of (n−c choose k) to (n choose k), then averages over tasks.
When n equals k, the formula simplifies: a task counts as solved if any of its attempts succeeded, and pass@k is the share of tasks solved at least once.
A worked example from CWE-bench v2
Collinear AI’s CWE-bench v2 has 100 public tasks and runs four attempts per task, so 400 trials in all. The reported pass@1 is the mean reward over those 400 trials, and pass@4 is the share of tasks with at least one successful attempt. For the leader, Gemini 4 Argon, Collinear reports pass@1 of 67.8% and pass@4 of 81.0%. Claude Opus 5.5 scores 67.3% and 78.0%.
Read together, they say Gemini 4 Argon succeeds on roughly two of three attempts, and on 81 of 100 tasks it succeeds at least once in four tries. The gap between the two numbers measures inconsistency: an agent with high pass@4 but lower pass@1 can solve problems but cannot do so reliably. Collinear also notes that even the leader fails 19% of tasks on every attempt.
Cost per rollout
This is the average spend for one attempt, including tokens for reasoning, tool calls and retries. In the CWE-bench v2 write-up, DeepSeek V4.1 Flash reaches 57.0% pass@1 at $0.04 per rollout, Claude Opus 5.5 adds about ten points at $1.83, and Gemini 4 Argon edges ahead at $3.92. A chart of score against cost lets you find the Pareto frontier, the set of models for which no other model is both cheaper and better.
Common traps
- Comparing pass@1 with pass@k. pass@4 will always be at least pass@1. Compare like with like.
- Ignoring n. pass@k estimates from very few samples are noisy.
- Self-reported numbers. Check who ran the evaluation and which harness they used.
- Contamination. Verifiers that reject the known upstream patch, as CWE-bench v2 does, reduce memorization.
- Ignoring cost and latency. A small score gain at ten times the price may not be worth it.
Quick reference table
| Term | Question it answers |
|---|---|
| pass@1 | How often does a single attempt work? |
| pass@k | How often does at least one of k attempts work? |
| Mean reward | What is the average score across all trials? |
| Cost per rollout | What does one attempt cost? |
| Held-out set | How does the model do on tasks it could not have seen? |
What to watch next
- Leaderboards that report both pass@1 and pass@k with cost.
- Wider use of held-out test sets to limit contamination.
- Standard reporting of confidence intervals on agent benchmarks.
Key takeaways
- pass@1 measures reliability; pass@k measures capability with retries.
- The gap between them shows how consistent an agent is.
- Always read cost per rollout beside accuracy.
- Check who ran the evaluation and how many attempts they used.
Frequently asked questions
What is the difference between pass@1 and pass@k?
pass@1 is the chance a single attempt succeeds, while pass@k is the chance that at least one of k attempts succeeds, so pass@k is never lower.
Where does pass@k come from?
It was defined in the 2021 paper “Evaluating Large Language Models Trained on Code,” which introduced HumanEval and an unbiased estimator for the metric.
Why report cost per rollout?
Because two agents with similar scores can differ in price by a factor of 100, and the cheaper one may be the better choice for many jobs.
