Evaluating Long-Context Reasoning in LLM-Based WebAgents

TL;DR

This work introduces a long-context benchmark for evaluating action-taking WebAgents operating on live Internet environments. It enforces sequentially dependent subtasks with noise-injected histories to stress memory, retrieval, and planning across contexts up to 150k tokens. The study shows dramatic performance degradation as context length grows, with most models falling below 10% success in long-context scenarios, and identifies loops and lost task objectives as key failure modes. An implicit RAG (iRAG) approach yields modest retrieval and task progress gains, but the results underscore the need for robust memory architectures, improved context filtering, and enhanced planning to enable reliable long-horizon WebAgent execution.

Abstract

As large language model (LLM)-based agents become increasingly integrated into daily digital interactions, their ability to reason across long interaction histories becomes crucial for providing personalized and contextually aware assistance. However, the performance of these agents in long context scenarios, particularly for action-taking WebAgents operating in realistic web environments, remains largely unexplored. This paper introduces a benchmark for evaluating long context reasoning capabilities of WebAgents through sequentially dependent subtasks that require retrieval and application of information from extended interaction histories. We develop a novel evaluation framework that simulates multi-session user interactions by injecting irrelevant task trajectories between dependent subtasks, creating contexts ranging from 25,000 to 150,000 tokens. Through extensive evaluation of four popular models, Claude-3.7, GPT-4.1, Llama 4, and o4-mini, we observe a dramatic performance degradation as context length increases, with success rates dropping from 40-50\% in baseline conditions to less than 10\% in long context scenarios. Our detailed error analysis reveals that agents primarily fail due to getting stuck in loops and losing track of original task objectives. We further propose an implicit RAG approach that provides modest improvements by generating task-relevant summaries, though fundamental limitations in long context reasoning persist. These findings highlight critical challenges for deploying WebAgents in realistic, long-term user interaction scenarios and provide insights for developing more robust agent architectures capable of maintaining coherent task execution across extended contexts.

Figures (11)

• Figure 1: Benchmark framework and evaluation overview. The left column shows the setup of the multi-session context. The middle column provides examples of the split subtasks as well as the (irrelevant) multi-session tasks. The right flowchart demonstrates benchmark and evaluation process for the trajectory the agent takes.
• Figure 2: Number of tasks injected at each context length
• Figure 3: Comparison of performance of baseline vs implicit RAG at 150k context length.
• Figure 4: Success rate of task completion at varying context lengths
• Figure 5: Breakdown of retrieval performance and task performance