LiveCodeBench was introduced in March 2024 by a ten-person team whose first author is Naman Jain. Its central idea is simple: if you keep collecting fresh problems and record when each one was published, you can always test a model on problems it could not have seen during training.
What it measures
The main task is competitive programming: reading a problem statement, designing an algorithm and writing code that passes hidden tests within time limits. Beyond plain code generation, LiveCodeBench also tests three related skills:
- self-repair: fixing a solution after seeing the error it produced;
- code execution: predicting what a given program will output;
- test output prediction: working out the expected output for a test input from the problem statement.
Together they give a broader view of coding ability than writing code alone.
How it works
Problems are collected from contests on three platforms: LeetCode, AtCoder and Codeforces. Each problem is tagged with its release date and a difficulty level, and solutions are checked by running them against test cases. The headline metric is pass@1, the share of problems solved on the first attempt.
The dataset grows in numbered releases. According to the official repository, release_v1 had 400 problems published between May 2023 and March 2024, and release_v6 had 1,055 problems published between May 2023 and April 2025. Evaluators can restrict scoring to a date window, for example only problems released after a model’s training cutoff. This is how the authors spotted contamination: the repository notes that DeepSeek models are scored only on problems released after August 2023 to counter it.
A typical problem might describe a grid with obstacles and ask for the number of shortest paths modulo a large prime, with input sizes large enough that a naive solution times out.
Why it matters
Static coding benchmarks like HumanEval leak into training data and saturate. LiveCodeBench’s rolling, time-stamped design makes contamination measurable and keeps the benchmark fresh. It became a standard line in model cards for reasoning models, whose step-by-step thinking helps a lot on algorithmic problems.
Limitations and criticisms
- Competitive programming is not software engineering. Contest problems reward algorithmic tricks under strict constraints, not maintaining large codebases. SWE-bench Verified and Terminal-Bench cover that ground.
- Version confusion. Scores depend on which release and date window were used. Numbers from different windows are not comparable.
- Difficulty mix. The share of easy, medium and hard problems changes with each window, which can shift scores.
- Archiving and saturation. Frontier reasoning models now score highly, and some independent evaluators, such as Vals AI, have archived their LiveCodeBench leaderboards.
Reading the scores
Always check the release version and date range. A model evaluated only on problems after its training cutoff gives the most honest result. Hard-problem accuracy is usually the most informative split for frontier models, since easy problems are close to solved. The official leaderboard on the project website lets you choose the time window; scores change as new problems are added, so check it directly for current results.