CodeTaste: Can LLMs Generate Human-Level Code Refactorings?

TL;DR

This paper investigates if LLM agents can execute refactorings reliably and identifies the refactorings that human developers actually chose in real codebases, and presents CodeTaste, a benchmark of refactoring tasks mined from large-scale multi-file changes in open-source repositories.

Abstract

Large language model (LLM) coding agents can generate working code, but their solutions often accumulate complexity, duplication, and architectural debt. Human developers address such issues through refactoring: behavior-preserving program transformations that improve structure and maintainability. In this paper, we investigate if LLM agents (i) can execute refactorings reliably and (ii) identify the refactorings that human developers actually chose in real codebases. We present CodeTaste, a benchmark of refactoring tasks mined from large-scale multi-file changes in open-source repositories. To score solutions, we combine repository test suites with custom static checks that verify removal of undesired patterns and introduction of desired patterns using dataflow reasoning. Our experimental results indicate a clear gap across frontier models: agents perform well when refactorings are specified in detail, but often fail to discover the human refactoring choices when only presented with a focus area for improvement. A propose-then-implement decomposition improves alignment, and selecting the best-aligned proposal before implementation can yield further gains. CodeTaste provides an evaluation target and a potential preference signal for aligning coding agents with human refactoring decisions in realistic codebases.

Figures (7)

• Figure 1: Overview of the CodeTaste benchmark pipeline. The workflow follows six key stages: (1) Commit Discovery from GitHub; (2) Generation of Task Descriptions; (3) Rule Generation of static analysis rules that capture refactoring intent; (4) Build Environment to run test suite and agent inference in containerized environment; (5) Inference where the agent generates a patch; and (6) Evaluation of generated patches against static analysis rules and test suite.
• Figure 2: Data funnel for obtaining the CodeTaste benchmark instances. In each step of the main commit pipeline, we filter for the top $n$ entries using different scoring schemes. Finally, we extract 100 instances for which our agentic setup produced a working execution environment.
• Figure 3: Example OpenGrep rule $\gamma \in \Gamma$ with symbolic propagation. In repository $R = \{f\}$, the two matching witnesses $\Omega(\gamma, R) = \{\omega_1,\omega_2\}$, include aliased and direct .colors property access. Even though match count $M(\gamma, R) = 3$, coverage $\mathcal{C}(\gamma, R) = 2$, as Line 4 contains several matches.
• Figure 4: Sketch of the evaluation metrics for CodeTaste.
• Figure 5: Programming language distribution in the selected $100$ instances of CodeTaste