SWE-Refactor: A Repository-Level Benchmark for Real-World LLM-Based Code Refactoring

TL;DR

SWE-Refactor addresses the need for realistic, semantics-preserving evaluation of LLM-based code refactoring by mining 1,099 pure Java refactorings (atomic and compound) from 18 projects and validating them through compilation, test execution, and AST-based detection. It introduces an automated four-step pipeline to ensure purity, enrich repository-level context, and verify behavior, while providing three evaluation metrics that combine functional correctness and human-likeness. Across nine LLMs, results show that atomic refactorings are more tractable than complex, multi-file transformations, with open-source models lagging behind general-purpose LLMs and multi-agent prompting offering the strongest gains for challenging cases. The work highlights the remaining gap in compound refactoring performance, demonstrates the value of repository-aware evaluation, and releases the dataset and results to support future research in LLM-assisted code maintenance and refactoring.

Abstract

Large Language Models (LLMs) have recently attracted wide interest for tackling software engineering tasks. In contrast to code generation, refactoring demands precise, semantics-preserving edits that improve program structure, which also makes automated evaluation challenging. However, existing refactoring benchmarks commonly suffer from three shortcomings: limited coverage of refactoring scenarios, the inclusion of instances that mix refactoring with unrelated changes, and insufficient repository-level context for realistic assessment. To mitigate these issues, we introduce SWE-Refactor, a new benchmark for LLM-based code refactoring. SWE-Refactor comprises 1,099 developer-written, behavior-preserving refactorings mined from 18 Java projects, including 922 atomic and 177 compound instances. Each instance is validated via compilation, test execution, and automated refactoring detection tools to ensure correctness. We evaluate nine widely used LLMs on SWE-Refactor, covering models such as GPT-4o-mini, DeepSeek-V3, and CodeLLaMa, to provide representative reference results. Our results show that complex and compound refactorings remain the primary source of failures; notably, an OpenAI Codex agent achieves only 39.4% success on compound instances. We release SWE-Refactor and all evaluation results to facilitate future research on LLM-based code refactoring.

Figures (4)

• Figure 1: An overview of the data in SWE-Refactor.
• Figure 2: Our Automated Pipeline to Construct SWE-Refactor.
• Figure 3: Comparison of successful refactorings.
• Figure 4: RAG Construction and Retrieval Pipeline.