Beyond Strict Rules: Assessing the Effectiveness of Large Language Models for Code Smell Detection

TL;DR

This paper addresses code-smell detection by evaluating four large language models on nine smells across 30 real-world Java projects, with ground-truth labels obtained from 76 developers evaluating 268 candidates. It shows that LLMs excel on structurally simple smells, while performance varies for more subjective smells; a voting-based ensemble combining LLMs and static-analysis tools often improves recall and $F1$-Score for several smells. The authors contribute an extended, jointly annotated dataset that links LLM detections, static-tool outputs, and human judgments, enabling replication and benchmarking. They conclude that the optimal detection strategy depends on the desired trade-off between recall and precision, and they outline practical guidance for researchers and practitioners along with directions for future work including expanded model sets and cross-language evaluation.

Abstract

Code smells are symptoms of potential code quality problems that may affect software maintainability, thus increasing development costs and impacting software reliability. Large language models (LLMs) have shown remarkable capabilities for supporting various software engineering activities, but their use for detecting code smells remains underexplored. However, unlike the rigid rules of static analysis tools, LLMs can support flexible and adaptable detection strategies tailored to the unique properties of code smells. This paper evaluates the effectiveness of four LLMs -- DeepSeek-R1, GPT-5 mini, Llama-3.3, and Qwen2.5-Code -- for detecting nine code smells across 30 Java projects. For the empirical evaluation, we created a ground-truth dataset by asking 76 developers to manually inspect 268 code-smell candidates. Our results indicate that LLMs perform strongly for structurally straightforward smells, such as Large Class and Long Method. However, we also observed that different LLMs and tools fare better for distinct code smells. We then propose and evaluate a detection strategy that combines LLMs and static analysis tools. The proposed strategy outperforms LLMs and tools in five out of nine code smells in terms of F1-Score. However, it also generates more false positives for complex smells. Therefore, we conclude that the optimal strategy depends on whether Recall or Precision is the main priority for code smell detection.

Figures (3)

• Figure 1: Steps of our study.
• Figure 2: Prompt template used for detecting code smells
• Figure 3: Prompt used to detect Large Class