GPQA (“Graduate-Level Google-Proof Q&A”) was published in November 2023 by David Rein and colleagues. Its Diamond subset, the hardest and cleanest slice, became one of the most cited science benchmarks for reasoning models.
What it measures
GPQA tests whether a model can answer very difficult questions in biology, physics and chemistry: the kind that a specialist with a relevant PhD can answer, but a clever outsider cannot work out even with unlimited time on the internet. It was designed partly for research on “scalable oversight”, the problem of how humans can supervise AI systems that know more than they do.
How it works
Domain experts wrote the questions and explanations. Each question was then attempted by other experts in the same field and by three skilled “non-experts”, PhD holders from other domains who were allowed to use the web. The paper reports that non-experts spent over 30 minutes per question on average and still reached only about 34% accuracy.
The paper releases three sets:
- GPQA Extended: 546 questions;
- GPQA Main: 448 questions;
- GPQA Diamond: 198 questions where both expert validators answered correctly (or the second clearly acknowledged a mistake) and most non-experts answered incorrectly.
Every question is multiple choice with four options, so random guessing gives 25%. Scores are reported as accuracy, often averaged over several runs because a 198-question set is small enough for noise to matter. A typical question might describe a multi-step organic synthesis and ask which product forms, or pose a quantum mechanics problem that needs both the right concept and careful calculation.
Why it matters
When it launched, GPQA was hard for AI: GPT-4 with few-shot chain-of-thought prompting scored between 38% and 40% across the three sets in the paper. Because the questions resist simple lookup, the benchmark became a favourite way to show progress in step-by-step scientific reasoning, and labs report it in nearly every frontier model launch.
Limitations and criticisms
- Saturation. Leading models now score above 90% on Diamond, well beyond what expert validators achieved, so the benchmark no longer separates the very best systems.
- Small size. With 198 questions, one question is worth about half a percentage point. Differences of a point or two are within noise.
- Label noise. The authors estimate that roughly 74% of questions in the extended set are uncontroversially correct by conservative standards, which means a small share of “errors” at the top may reflect the answer key rather than the model.
- Contamination risk. The data ships with a canary string so that it can be filtered from training corpora, but that depends on everyone respecting it.
- Multiple choice. Picking among four answers can be easier than solving the problem cold.
Reading the scores
In the paper, expert validators reached about 65% on the main set (74% when discounting mistakes they later recognised), and non-experts about 34%. A score above 90% is now typical of frontier models.
The top models listed above come from Artificial Analysis, which runs GPQA Diamond independently on many models; the page listed three leading entries when checked on 23 October 2026. Labs’ self-reported numbers may differ because of prompting and sampling choices.