Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpouthakoun, Bardia Koopah, Changran Hu, Etash Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Aggarwal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjörn Kolbeinsson, Jesse Hu, Christopher Michael Rytting, Ryan Marten, Yixin Wang, Alex Dimakis, Andy Konwinski, Ludwig Schmidt

TL;DR

Terminal-Bench introduces a hard, realistic benchmark suite (Terminal-Bench 2.0) for evaluating agentic LLMs operating in command line environments. It defines a rich task format with instruction, containerized context, verification tests, and a human-authored oracle solution, and validates 89 tasks selected from a larger collection through rigorous review. The empirical results show frontier models achieve under 65% task resolution, with substantial cross-model variance and notable cost and time dynamics, underscoring both the progress and remaining gaps in autonomous terminal problem-solving. An extensive error taxonomy and per-command failure analysis illuminate concrete avenues for improvement, while Harbor and Terminus 2 provide scalable, reproducible infrastructure for future benchmarking. The dataset and evaluation harness are released to foster ongoing advancement in realistic, long-horizon agentic capabilities in terminal settings.

Abstract

AI agents may soon become capable of autonomously completing valuable, long-horizon tasks in diverse domains. Current benchmarks either do not measure real-world tasks, or are not sufficiently difficult to meaningfully measure frontier models. To this end, we present Terminal-Bench 2.0: a carefully curated hard benchmark composed of 89 tasks in computer terminal environments inspired by problems from real workflows. Each task features a unique environment, human-written solution, and comprehensive tests for verification. We show that frontier models and agents score less than 65\% on the benchmark and conduct an error analysis to identify areas for model and agent improvement. We publish the dataset and evaluation harness to assist developers and researchers in future work at https://www.tbench.ai/ .

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

TL;DR

Terminal-Bench introduces a hard, realistic benchmark suite (Terminal-Bench 2.0) for evaluating agentic LLMs operating in command line environments. It defines a rich task format with instruction, containerized context, verification tests, and a human-authored oracle solution, and validates 89 tasks selected from a larger collection through rigorous review. The empirical results show frontier models achieve under 65% task resolution, with substantial cross-model variance and notable cost and time dynamics, underscoring both the progress and remaining gaps in autonomous terminal problem-solving. An extensive error taxonomy and per-command failure analysis illuminate concrete avenues for improvement, while Harbor and Terminus 2 provide scalable, reproducible infrastructure for future benchmarking. The dataset and evaluation harness are released to foster ongoing advancement in realistic, long-horizon agentic capabilities in terminal settings.

Abstract

AI agents may soon become capable of autonomously completing valuable, long-horizon tasks in diverse domains. Current benchmarks either do not measure real-world tasks, or are not sufficiently difficult to meaningfully measure frontier models. To this end, we present Terminal-Bench 2.0: a carefully curated hard benchmark composed of 89 tasks in computer terminal environments inspired by problems from real workflows. Each task features a unique environment, human-written solution, and comprehensive tests for verification. We show that frontier models and agents score less than 65\% on the benchmark and conduct an error analysis to identify areas for model and agent improvement. We publish the dataset and evaluation harness to assist developers and researchers in future work at https://www.tbench.ai/ .
Paper Structure (59 sections, 36 figures, 5 tables)

This paper contains 59 sections, 36 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Terminal-Bench
  3. Task Formulation
  4. Dataset Construction
  5. Verification
  6. Composition
  7. Experimental Setup
  8. Terminus 2
  9. Agents
  10. Models
  11. Harbor
  12. Results
  13. Cost and Model Performance
  14. Model Performance Over Time
  15. Predicted vs. Empirical Difficulty
  16. ...and 44 more sections

Figures (36)

  • Figure 1: Task resolution rate per model on Terminal-Bench 2.0. The error bars correspond to a 95% confidence interval. The agent scaffold used to report each model was chosen to maximize performance. Results for all agents and models evaluated are in \ref{['app:detailed_results']}.
  • Figure 2: A Terminal-Bench task is composed of an instruction, a Dockerfile, a set of tests, and an oracle solution. Agents run inside a container into which the tests are copied and executed.
  • Figure 3: Our task audit process consists of multiple rounds of manual review of each task by several project contributors to audit for common mistakes. Between the three reviews, the average task received approximately three hours of combined reviewer attention, implying multiple hundreds of person-hours went into reviewing alone, excluding the time spent creating the tasks.
  • Figure 4: Tasks per category in Terminal-Bench 2.0. Categories were assigned by the task author. Software engineering is the largest category, although no single category represents the majority of tasks. Terminal-Bench 2.0 has representation across a variety of categories, including non-engineering-specific categories such as "personal assistant" and "video processing".
  • Figure 5: The Pareto frontier of agent performance showing the tradeoff between performance and cost (log scale) on Terminal-Bench 2.0.
  • ...and 31 more figures