From Human to Machine Refactoring: Assessing GPT-4's Impact on Python Class Quality and Readability

TL;DR

This paper empirically evaluates GPT-4o-driven, class-level refactoring of Python code using Fowler-inspired techniques on the ClassEval benchmark. It adopts a multi-perspective framework—checking behavioral correctness with unit tests, assessing code quality via static analysis tools, and measuring readability with a dedicated metric. Results show that GPT-4o can produce behavior-preserving refactorings that reduce code smells and improve certain quality metrics, yet readability often declines, highlighting a trade-off between structural quality and human understandability. The study provides a replication package and points to hybrid approaches that combine LLM capabilities with formal analysis and testing to realize practical, reliable automated refactoring workflows.

Abstract

Refactoring is a software engineering practice that aims to improve code quality without altering program behavior. Although automated refactoring tools have been extensively studied, their practical applicability remains limited. Recent advances in Large Language Models (LLMs) have introduced new opportunities for automated code refactoring. The evaluation of such an LLM-driven approach, however, leaves unanswered questions about its effects on code quality. In this paper, we present a comprehensive empirical study on LLM-driven refactoring using GPT-4o, applied to 100 Python classes from the ClassEval benchmark. Unlike prior work, our study explores a wide range of class-level refactorings inspired by Fowler's catalog and evaluates their effects from three complementary perspectives: (i) behavioral correctness, verified through unit tests; (ii) code quality, assessed via Pylint, Flake8, and SonarCloud; and (iii) readability, measured using a state-of-the-art readability tool. Our findings show that GPT-4o generally produces behavior-preserving refactorings that reduce code smells and improve quality metrics, albeit at the cost of decreased readability. Our results provide new evidence on the capabilities and limitations of LLMs in automated software refactoring, highlighting directions for integrating LLMs into practical refactoring workflows.

Figures (7)

• Figure 1: Distribution of the mean values of the number of refactorings per task across 10 runs.
• Figure 2: Comparison of two versions for a snippet in the dataset: on the left, the refactored version by GPT-4o, while on the right, the canonical solution by ClassEval. Unlike the canonical solution, the refactored version does not pass the tests.
• Figure 3: Distribution of metric differences across ten GPT-4o refactoring runs. Each group of boxes corresponds to a metric, and each color represents a distinct iteration. Negative values indicate smaller or less complex refactored code.
• Figure 4: Per-iteration Wilcoxon effect sizes (r) comparing refactored and canonical code across all static analysis metrics. Each cell represents the effect size for a specific metric (row) and GPT-4o refactoring run (column). Negative values (darker colors) indicate improvements, meaning the refactored code exhibits fewer detected issues.
• Figure 5: Comparison between the canonical (right) and refactored (left) implementations of a fragment from ClassEval_57 task. This refactoring eliminates redundant logic, simplifies control flow, and contributes to the reduction of Flake8 warnings observed in the static analysis.