MMLU-Pro was released in June 2024 by a team led by Yubo Wang, Xueguang Ma and Ge Zhang, from TIGER-Lab and collaborators. It was built because the original MMLU had become too easy and too noisy to tell frontier models apart.
What it measures
Like its predecessor, MMLU-Pro tests broad academic knowledge, but it deliberately shifts the balance toward reasoning. Many questions require multi-step calculation or analysis rather than recall of a single fact. The goal is a test where careful reasoning, not lucky guessing or memorisation, determines the score.
How it works
The dataset card lists 12,032 questions in 14 categories: maths, physics, chemistry, law, engineering, economics, health, psychology, business, biology, philosophy, computer science, history and “other”. The questions come from four sources:
- original MMLU questions, with trivial and ambiguous items removed;
- problems from STEM websites;
- TheoremQA;
- SciBench, a set of college-level science exam questions.
The biggest change is the answer format. Each question has up to ten options instead of four (GPT-4 was used to generate extra plausible distractors, which were then reviewed by humans). That cuts the chance of a correct random guess from 25% to 10%. Models are usually evaluated with chain-of-thought prompting, writing out their reasoning before choosing a letter, and the score is accuracy.
A typical item might be a physics problem that needs two or three calculation steps, where several of the ten options match common mistakes.
Why it matters
According to the paper, moving from MMLU to MMLU-Pro lowered model accuracy by 16 to 33 percentage points, which restored room at the top of the scale. The authors also found the benchmark is more stable: across 24 prompt styles, scores varied by about 2%, compared with 4–5% on MMLU. And chain-of-thought helped on MMLU-Pro while it did not on the original, which suggests the new questions really do reward reasoning. That made MMLU-Pro a standard line in model cards and in independent evaluations through 2024 and 2025.
Limitations and criticisms
- Saturation, again. Frontier models have climbed quickly on MMLU-Pro too. Some independent evaluators, such as Vals AI, have since archived their MMLU-Pro leaderboards.
- Inherited issues. Part of the dataset comes from MMLU, so some of the original’s quirks and its public exposure carry over.
- Generated distractors. Wrong options written with a model’s help can contain hints or errors that other models learn to exploit.
- Still multiple choice. Choosing among options is easier than producing an answer from scratch, and it does not test coding, writing or tool use.
Reading the scores
Random guessing gives about 10%. The project’s original mini-leaderboard listed GPT-4o at 72.55%, well below its MMLU score. Today, scores in the high 80s are common among frontier models, so small differences near the top should not be over-interpreted.
Scores change quickly and depend on prompting and answer extraction, so compare models on the same evaluator. The official repository links to the maintainers’ leaderboard. For a wider view of why tests like this saturate, read how AI benchmarks work and why they saturate.