AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rollouts

TL;DR

This paper addresses the gap in evaluating long-context autonomous agents by moving beyond static retrieval benchmarks to dynamic agent–environment interactions. It introduces AgentLongBench, a controllable benchmark built on deterministic environment rollouts using Lateral Thinking Puzzles, with two settings (Knowledge-Intensive and Knowledge-Free), two response formats (Concise and Verbose), and context lengths from $32K$ to $4M$ tokens. Experiments across proprietary, open-weight, and memory-augmented baselines reveal that although some models maintain performance with large contexts, many fail to sustain long-horizon reasoning; memory augmentation often provides little to no benefit, and the performance bottleneck shifts to the minimum token requirement for evidence localization in dense tool logs. The benchmark provides a transparent, extensible platform for diagnosing failure modes in long-context reasoning and guiding future design of robust tool-grounded agents for real-world workflows.

Abstract

The evolution of Large Language Models (LLMs) into autonomous agents necessitates the management of extensive, dynamic contexts. Current benchmarks, however, remain largely static, relying on passive retrieval tasks that fail to simulate the complexities of agent-environment interaction, such as non-linear reasoning and iterative feedback. To address this, we introduce \textbf{AgentLongBench}, which evaluates agents through simulated environment rollouts based on Lateral Thinking Puzzles. This framework generates rigorous interaction trajectories across knowledge-intensive and knowledge-free scenarios. Experiments with state-of-the-art models and memory systems (32K to 4M tokens) expose a critical weakness: while adept at static retrieval, agents struggle with the dynamic information synthesis essential for workflows. Our analysis indicates that this degradation is driven by the minimum number of tokens required to resolve a query. This factor explains why the high information density inherent in massive tool responses poses a significantly greater challenge than the memory fragmentation typical of long-turn dialogues.

Figures (14)

• Figure 1: The distribution of distinct question types under the Knowledge-Intensive setting with Concise-Response formatting in our dataset.
• Figure 2: Overview of the Data Construction Pipeline for AgentLongBench. The dataset is constructed by simulating Environment Rollouts (Top), which capture the iterative interaction logs between an LLM agent, tools, and environmental feedback. These trajectories are then used to derive three categories of QA tasks (Left): QA in Tool Response, QA in Environment Response, and Final Guess. The construction process incorporates two data settings (Right Top) to distinguish between Knowledge-Intensive and Knowledge-Free scenarios, and two tool response formats (Right Bottom)—Concise vs. Verbose—to simulate different context densities and noise levels.
• Figure 3: Main Results on Knowledge-Intensive & Concise-Response Setting. The heatmap visualizes model performance across varying context lengths (32K to 2M). Green indicates higher accuracy.
• Figure 4: Main Results on Knowledge-Free & Concise-Response Setting.
• Figure 5: Main Results on Knowledge-Intensive & Verbose-Response Setting.