AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts

TL;DR

AgencyBench addresses the need for evaluating autonomous agents in long-horizon, real-world contexts by introducing a comprehensive benchmark of 6 capabilities across 32 authentic scenarios and 138 tasks, each defined by queries, deliverables, and rubrics. The authors build an end-to-end automated evaluation pipeline that uses a dedicated agent scaffold, a user simulation agent, and a Docker-based remote sandbox to produce machine-checkable artifacts and rubric-based scores, achieving scalable rollout collection without human-in-the-loop supervision. Their large-scale experiments reveal a persistent gap between closed-source and open-source models in overall performance and efficiency, with distinct tool-use preferences and self-correction abilities; they also show strong ecosystem effects where models perform best within their native scaffolds. The work highlights the importance of co-optimizing model architecture with agentic frameworks and provides a public toolkit to drive further research toward resource-efficient, self-correcting, scaffold-agnostic autonomous agents that deliver real-world utility.

Abstract

Large Language Models (LLMs) based autonomous agents demonstrate multifaceted capabilities to contribute substantially to economic production. However, existing benchmarks remain focused on single agentic capability, failing to capture long-horizon real-world scenarios. Moreover, the reliance on human-in-the-loop feedback for realistic tasks creates a scalability bottleneck, hindering automated rollout collection and evaluation. To bridge this gap, we introduce AgencyBench, a comprehensive benchmark derived from daily AI usage, evaluating 6 core agentic capabilities across 32 real-world scenarios, comprising 138 tasks with specific queries, deliverables, and rubrics. These scenarios require an average of 90 tool calls, 1 million tokens, and hours of execution time to resolve. To enable automated evaluation, we employ a user simulation agent to provide iterative feedback, and a Docker sandbox to conduct visual and functional rubric-based assessment. Experiments reveal that closed-source models significantly outperform open-source models (48.4% vs 32.1%). Further analysis reveals significant disparities across models in resource efficiency, feedback-driven self-correction, and specific tool-use preferences. Finally, we investigate the impact of agentic scaffolds, observing that proprietary models demonstrate superior performance within their native ecosystems (e.g., Claude-4.5-Opus via Claude-Agent-SDK), while open-source models exhibit distinct performance peaks, suggesting potential optimization for specific execution frameworks. AgencyBench serves as a critical testbed for next-generation agents, highlighting the necessity of co-optimizing model architecture with agentic frameworks. We believe this work sheds light on the future direction of autonomous agents, and we release the full benchmark and evaluation toolkit at https://github.com/GAIR-NLP/AgencyBench.

Figures (5)

• Figure 1: Overview of AgencyBench.Left: Distribution of the 32 scenarios and 138 tasks across 6 distinct agentic capabilities. Right: Comparison with existing benchmarks. AgencyBench focuses on diverse, long-horizon real-world tasks, requiring an average of 1M tokens and 90 multi-turn tool uses. It integrates a user simulation agent for iterative feedback and a Docker-based sandbox for automated rubric-based assessment.
• Figure 2: AgencyBench Rollout Generation and Evaluation Pipeline. Rollout generation takes place within workspace, where the agent receives task queries and deliverables, completing tasks through multi-turn interactions with the environment (e.g., tool execution results and feedback from the user simulation agent). Upon task completion, deliverables are synced to a Docker sandbox for operation execution (e.g., UI actions), and resulting artifacts are transferred to eval-space for scoring ($0-10$) via rule-based or LLM-based judges based on task rubrics.
• Figure 3: An Illustrative Evaluation Scenario in AgencyBench: Developing a Gomoku Game. The scenario consists of five sequential tasks with increasing complexity, requiring the incremental addition of new features. The primary deliverables include HTML, CSS, and JS source code. Evaluation scripts execute these files within a remote Docker sandbox, performing interactive operations such as clicking, screen recording, and capturing screenshots (visualized as video frames in the figure). The resulting evaluation artifacts are retrieved to eval-space, where text and vision agents assess the code and visual deliverables, respectively, providing scores and qualitative feedback based on rubrics. Right: The file organization architecture during runtime, showing the isolated workspace and eval-space for each task to ensure environmental consistency and prevent cross-task interference.
• Figure 4: Efficiency Comparison Across Models. Efficiency is calculated by dividing the average score by the number of attempts and average token consumption, respectively. GPT-5.2 achieves the highest attempt efficiency, while Qwen-3-235B-A22B-Thinking ranks the lowest. For token efficiency, Grok-4.1-Fast performs best, whereas Claude-4.5-Sonnet is the least efficient one.
• Figure 5: Tool Invocation Patterns Across Models. Claude-4.5-Opus and GPT-5.2 shows a preference for shell execution tools, while Gemini-3-Pro and Qwen-3-235B-A22B-Thinking favor file operation and memory management. Grok-4.1-Fast, GLM-4.6, and Deepseek-V3 series exhibit a strong preference for web search tools.