RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code

TL;DR

RefactorBench introduces a real-world, multi-hop code-refactoring benchmark composed of 100 handcrafted tasks across 9 Python repos, evaluated with AST-based checks to emphasize structural correctness over exact line matches. The study reveals substantial gaps in current LM agents’ ability to perform multi-file reasoning and track past actions, with base instruction success around 22% and human performance around 87%. Through trajectory analyses, the authors identify key failure modes and demonstrate that conditioning agents on explicit state representations yields substantial gains (up to 43.9% average improvement, 71% on subtasks) and generalizes toward state-aware environments via a state update policy. The work argues for robust stateful reasoning as essential for scaling language agents in real-world software engineering tasks and outlines directions for future research in state-aware interfaces and long-horizon planning.

Abstract

Recent advances in language model (LM) agents and function calling have enabled autonomous, feedback-driven systems to solve problems across various digital domains. To better understand the unique limitations of LM agents, we introduce RefactorBench, a benchmark consisting of 100 large handcrafted multi-file refactoring tasks in popular open-source repositories. Solving tasks within RefactorBench requires thorough exploration of dependencies across multiple files and strong adherence to relevant instructions. Every task is defined by 3 natural language instructions of varying specificity and is mutually exclusive, allowing for the creation of longer combined tasks on the same repository. Baselines on RefactorBench reveal that current LM agents struggle with simple compositional tasks, solving only 22% of tasks with base instructions, in contrast to a human developer with short time constraints solving 87%. Through trajectory analysis, we identify various unique failure modes of LM agents, and further explore the failure mode of tracking past actions. By adapting a baseline agent to condition on representations of state, we achieve a 43.9% improvement in solving RefactorBench tasks. We further extend our state-aware approach to encompass entire digital environments and outline potential directions for future research. RefactorBench aims to support the study of LM agents by providing a set of real-world, multi-hop tasks within the realm of code.

Figures (8)

• Figure 1: RefactorBench task instances are multi-file refactors verified by custom AST unit testing. Tasks are split over popular open-source Python repositories and reference solutions are withheld to prevent overfitting to the task.
• Figure 2: Distribution of RefactorBench tasks across 9 open source Python repositories.
• Figure 3: Baselines of a prompt adjusted SWE-agent with gpt-4 and a human developer given IDE access and a time limit of 5 minutes.
• Figure 4: GPT-4 Turbo accuracy in end state construction drops consistently over increased iterations of actions. Prompts and examples are available in Appendix \ref{['appendix:e']}.
• Figure 5: Comparison of a prompt adjusted SWE-agent and a state-aware SWE-agent implementation, both using gpt-4. Sample implementation code available in Appendix \ref{['appendix:g']}.