Putting It All into Context: Simplifying Agents with LCLMs

TL;DR

The paper tackles the problem of increasingly complex LM agent architectures by asking whether a fully observable task like SWE-Bench can be solved with a state-in-context approach using long-context LLMs. It introduces DirectSolve, which prompts a long-context LM to solve problems from the entire (or compressed) environment state, and SelectSolve, a two-stage hybrid that localizes files with an LCLM and then repairs with a short-context LM. Empirical results on SWE-Bench Verified show DirectSolve with Gemini-2.5-Pro achieving about 50.8% pass@1 and 60.2% pass@8, while a Claude-3.7-Sonnet–assisted SelectSolve reaches 48.6% pass@1; these figures indicate that scaffolding-free, monolithic LCLM agents can rival heavily engineered pipelines, although stronger coding performance remains an advantage for specialized tooling. The findings suggest that as LCLMs improve in capability and context length, simpler monolithic architectures could become practical and cheaper by reducing the need for task-specific scaffolding, with potential broad implications for memory, retrieval, and agent design in software engineering tasks.

Abstract

Recent advances in language model (LM) agents have demonstrated significant potential for automating complex real-world tasks. To make progress on these difficult tasks, LM agent architectures have become increasingly complex, often incorporating multi-step retrieval tools, multiple agents, and scaffolding adapted to the underlying LM. In this work, we investigate whether all of this complexity is necessary, or if parts of these scaffolds can be removed on challenging tasks like SWE-bench. We show that in the case of SWE-bench, simply putting the entire environment into the context of a long context language model (LCLM) and properly prompting the model makes it competitive with carefully tuned, complex agent scaffolds. We show that a Gemini-1.5-Pro model without any scaffolding or tools achieves 38% on SWE-Bench-Verified, comparable with approaches using carefully tuned agent scaffolds (32%). While the unscaffolded approach with Gemini-1.5-Pro falls short of the strongest agentic architectures, we demonstrate that the more capable Gemini-2.5-Pro using the same unscaffolded approach directly attains a 50.8% solve rate. Additionally, a two-stage approach combining Gemini-1.5-Pro with Claude-3.7 achieves a competitive 48.6% solve rate.

Figures (4)

• Figure 1: Overview. (Left) Traditional agent design treats the environment as partially observable and curates elaborate scaffolding (such as tools and execution pipelines) upon LMs to collect the necessary information for solving the task. (Right) In contrast, we leverage the increasingly powerful capabilities of long-context LMs to develop state-in-context agents that eliminate the need for complex scaffolding. These agents achieve full observability by maintaining the entire environment state within the context of LMs, turning open-ended agentic tasks into direct, close-ended tasks where LMs excel.
• Figure 2: Instantiation of state-in-context agents for software engineering. (Left) When the environment state (i.e., code repository) exceeds the context length limit of LCLMs, we apply a simple compression step that ranks files by their relevance and selectively includes files up to the maximum context limit. (Right) We instatiate state-in-context agents in two ways: DirectSolve directly generates the solution using LCLMs that consume the entire (compressed) state, which are then fed into short-context Language Models (SCLMs) to generate the solutions, leveraging the superior problem-solving capabilities of SCLMs.
• Figure 3: Codebase token distributions and context window compatibility. (Left) Percentage of problem instances within the 2M token limit of Gemini-1.5-Pro across three state representation methods: all files, core files only (test and non-python files removed), and core code only (documentation further removed) in SWE-Bench-Verified. (Right) Distribution of token counts for code-only content of instances in SWE-Bench-Verified. The dashed line marks the 2M token threshold, demonstrating that most codebases can fit within current LCLMs context constraints through selective content filtering.
• Figure 4: Model performance vs. Context Length for DirectSolve with target files placed at the front of the prompt.