Evaluating the Effectiveness of Small Language Models in Detecting Refactoring Bugs
Rohit Gheyi, Marcio Ribeiro, Jonhnanthan Oliveira
TL;DR
This study investigates whether Small Language Models (SLMs) can reliably detect bugs arising from code refactorings in IDEs, focusing on Type I (incorrect transformations) and Type II (valid refactorings blocked). Using zero-shot prompting across eight SLMs on 100 bugs from Java and Python (16 refactoring types), Phi-4 14B and o3-mini-high emerge as the strongest performers for Type I, achieving 71.4% and 84.3% detection respectively, while Type II detection lags with a maximum around 40%. The approach demonstrates cross-language generalizability and suggests a practical, cost-effective hybrid workflow that leverages open-source models for first-pass validation and proprietary models for high-assurance checks, with metamorphic testing indicating robustness against code structure variations. These findings highlight the potential of SLMs to streamline refactoring validation and motivate further development of mixed-model strategies and integration into IDEs and CI pipelines. The work also underscores the ongoing need for human oversight to validate model explanations and handle the more challenging Type II cases.
Abstract
Popular IDEs frequently contain bugs in their refactoring implementations. Ensuring that a transformation preserves a program's behavior is a complex task. Traditional detection methods rely on predefined preconditions for each refactoring type, limiting their scalability and adaptability to new transformations. These methods often require extensive static and dynamic analyses, which are computationally expensive, time-consuming, and may still fail to detect certain refactoring bugs. This study evaluates the effectiveness of Small Language Models (SLMs) in detecting two types of refactoring bugs in Java and Python: (i) transformations that introduce errors or behavioral changes (Type I) and (ii) transformations unnecessarily blocked by IDEs despite being valid (Type II). We assess whether Llama 3.2 3B, Mistral 7B, Gemma 2 9B, Gemma 3 12B, DeepSeek-R1 14B, Phi-4 14B, o1-mini, and o3-mini-high can accurately detect 100 refactoring bugs reported in widely used Java and Python IDEs, such as Eclipse and NetBeans. The study covers 16 refactoring types and employs zero-shot prompting on consumer-grade hardware to evaluate the models' ability to reason about refactoring correctness without explicit prior training. The proprietary o3-mini-high model achieved the highest detection rate, identifying 84.3% of Type I bugs. The open-source Phi-4 14B performed comparably well, demonstrating strong effectiveness across both bug types. However, o3-mini-high struggled with Type II bugs, correctly identifying and applying valid but blocked transformations in only 40% of cases. The findings highlight the potential of SLMs for efficiently detecting refactoring bugs, particularly in verifying behavioral changes. Additionally, SLMs offer a more adaptable solution capable of generalizing across different refactoring types and programming languages, addressing key limitations of traditional approaches.