Collinear AI has opened a public set of 100 tasks called CWE-bench v2, a dataset for testing whether AI agents can find and fix security weaknesses in real open-source code. According to the November 3 announcement on Hugging Face, the best of 11 evaluated models, Gemini 4 Argon, passes 67.8% of tasks on a single attempt. The data is not peer-reviewed research; it is a company-published benchmark.
What the dataset contains
Each task gives an agent a checkout of a real open-source repository and a short instruction. The goal is to find and fix a security weakness without breaking existing functionality. The 100 public tasks cover 91 different CWEs. CWE, the Common Weakness Enumeration maintained by MITRE, is a community catalog of software weakness types such as injection flaws or improper input validation, so a model needs to handle many vulnerability classes rather than recognize one familiar pattern.
Every task is a self-contained Harbor task: a Dockerized environment, an instruction and a deterministic verifier. The dataset ships a registry.json, so the Harbor tool can run the whole benchmark directly from the Hugging Face repository. Scoring is fully programmatic, which makes results reproducible without extra cost. Collinear also keeps a held-out set of tasks no model has seen, available on request for evaluating a model.
How the tasks were checked
Collinear says it curated the programmatic checks with panels of LLM judges, pushing them toward hardness and fairness, and had each task reviewed by hand. For every task the team verified three things: that the instruction reads like a realistic request rather than an evaluation prompt, that the verifier checks correct behavior and not one specific shape of fix, and that the verifier fails memorized solutions such as the known upstream patch.
The last check addresses a common weakness of code benchmarks. If a fix already exists in a public repository, a model may reproduce it from training data. Rejecting the upstream patch forces models to solve the problem rather than recall the answer.
What the first results show
Of the 11 models evaluated, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, just ahead of Claude Opus 5.5 at 67.3% and 78.0%. Ninety-five of the 100 tasks are solved by at least one model, which suggests the tasks are solvable, but even the leader fails 19% of the set on every attempt.
Cost changes the picture. Collinear reports that DeepSeek V4.1 Flash, Claude Opus 5.5 and Gemini 4 Argon form the cost-performance frontier: DeepSeek reaches 57.0% pass@1 at $0.04 per rollout, Opus adds about ten points at $1.83, and Gemini edges ahead at $3.92. These figures are the publisher’s own, and they come from one company’s harness. See our cheat sheet on reading benchmark claims for questions to ask before relying on them.
How to use it
The dataset is gated: users accept its access conditions on the dataset page, then run Harbor with the repository URL, the number of attempts (-k 4 for four) and a model. The reported score is the mean reward over all 400 trials. The tasks are also tagged as reinforcement-learning environments, so the same environments and verifiers can supply rewards when training defensive agents. Open-weight models can add results by opening a pull request with a results file, while API-only models are added through the leaderboard Space.
For readers building their own evaluations, our guide to free dataset sources for AI projects covers where else to look, and our overview of AI in cybersecurity explains why defensive benchmarks matter.
What to watch next
- More models added to the leaderboard, especially open-weight systems.
- Whether the held-out set shows a gap compared with the public 100 tasks.
- Independent reproductions of the pass@1 and cost numbers.
- Whether the tasks are used as training environments for defensive agents.
- Updates to the access conditions on the gated dataset.
Key takeaways
- CWE-bench v2 offers 100 public tasks covering 91 weakness types, with deterministic verifiers.
- Gemini 4 Argon leads at 67.8% pass@1 and 81.0% pass@4, per Collinear.
- Cost per rollout ranges from $0.04 to $3.92 among the frontier models.
- The dataset is gated, and the results are self-reported by the publisher.
Frequently asked questions
What is CWE-bench v2?
It is a public set of 100 security tasks built from real open-source repositories, where an AI agent must fix a weakness without breaking functionality. Tasks span 91 CWE types.
Is CWE-bench v2 free to use?
The tasks are open but gated: you must accept the access conditions on the Hugging Face dataset page before running them.
What does pass@1 mean here?
It is the average share of tasks solved per attempt. Collinear runs four attempts per task and averages the reward over all 400 trials.
