Terminal-Bench is hosted by Stanford, the Laude Institute and the Harbor evaluation framework. Its 2.0 version was described in a January 2026 paper, “Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces”, with Mike A. Merrill, Alexander G. Shaw and Nicholas Carlini as the first of 85 authors; the paper was accepted at ICLR 2026. The current version is Terminal-Bench 4.0.
What it measures
Many real tasks given to AI agents happen in a terminal: compiling software, configuring servers, training a model, debugging a failing build or analysing data with command-line tools. Terminal-Bench measures whether an agent can carry out such work end to end, deciding which commands to run, reading their output and recovering from mistakes, without a human stepping in.
How it works
Each task consists of:
- a containerised environment (a sandboxed Linux system with the relevant files and software);
- an instruction describing what to achieve;
- a set of tests that check whether the job was done;
- a reference solution written by hand to prove the task is solvable.
The agent works inside the container until it decides it is finished, and the tests then decide pass or fail. The score is the percentage of tasks passed. Terminal-Bench 2.0 contained 89 tasks, each manually verified by three human reviewers; at publication, frontier models and agents scored below 65%. According to Artificial Analysis, Terminal-Bench 4.0 has 66 tasks spanning software, machine learning, science, operations, security, hardware and media, recalibrates compute and time allowances, and passes a task only if all of its tests pass.
A typical task might ask the agent to get a legacy C project compiling with a modern toolchain, or to recover data from a corrupted database file and write a summary to a specified path.
Why it matters
Terminal-Bench is one of the most direct tests of practical agentic ability: it rewards planning, tool use and persistence over long sequences of actions, the skills that matter for coding agents and automation. Because the same tasks can be run with different agent frameworks, it is also used to compare harnesses, not just models.
Limitations and criticisms
- Model plus harness. Scores reflect both the model and the agent software around it; leaderboard entries name both, and they should be compared with that in mind.
- Small task sets. With 66 to 89 tasks, each task is worth more than a percentage point, so close scores are effectively tied.
- Versions are not comparable. Versions 2.0, 2.1 and 4.0 use different tasks or settings. Independent evaluators such as Vals AI have archived their 2.x leaderboards.
- Time and cost limits. Changing the allowed time or compute can move results substantially, which version 4.0 tries to standardise.
Reading the scores
The top models above come from Artificial Analysis’s independent run of Terminal-Bench 4.0, which listed Kimi K3.1 Pro (67.2%), Gemini 4 Carbon (High) (65.2%) and Claude Sonnet 5.5 (Max) (63.6%) as the three leading entries when we checked on 15 November 2026; the page does not show its own update date. The labels in brackets are reasoning-effort settings, and Artificial Analysis notes a default fallback configuration. The official Terminal-Bench site hosts its own leaderboard, now for version 4.1, that ranks agent and model combinations; on 15 November 2026 its top two entries, Claude Code with Opus 5.5 and Claude Code with Fable 5.2, were still tied at 65.8%, ahead of Antigravity CLI with Gemini 4 Carbon (High) at 64.5%. Since even the best systems still fail roughly a third or more of the tasks, the benchmark has room left; check the official site for the latest results.