Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Propense Are Large Language Models at Producing Code Smells? A Benchmarking Study

Alejandro Velasco, Daniel Rodriguez-Cardenas, Luftar Rahman Alif, David N. Palacio, Denys Poshyvanyk

TL;DR

This work addresses the gap that traditional accuracy metrics fail to capture when evaluating code generation by LLMs. It introduces CodeSmellEval, consisting of the Propensity Smelly Score (PSC), the CodeSmellData dataset, and a protocol for estimating smell propensity from logits in decoder-based transformers. A case study with CodeLlama and Mistral demonstrates that both models exhibit substantial propensity to generate code smells, with 10 of 13 smells surpassing a propensity threshold of $0.5$ and distinct distributions across smells. The benchmark provides an interpretable, model-agnostic framework and a curated data resource to improve evaluation of code quality beyond accuracy, guiding future mitigation and research into the causes and remedies of smell generation in code produced by LLMs.

Abstract

Large Language Models (LLMs) have shown significant potential in automating software engineering tasks, particularly in code generation. However, current evaluation benchmarks, which primarily focus on accuracy, fall short in assessing the quality of the code generated by these models, specifically their tendency to produce code smells. To address this limitation, we introduce CodeSmellEval, a benchmark designed to evaluate the propensity of LLMs for generating code smells. Our benchmark includes a novel metric: Propensity Smelly Score (PSC), and a curated dataset of method-level code smells: CodeSmellData. To demonstrate the use of CodeSmellEval, we conducted a case study with two state-of-the-art LLMs, CodeLlama and Mistral. The results reveal that both models tend to generate code smells, such as simplifiable-condition and consider-merging-isinstance. These findings highlight the effectiveness of our benchmark in evaluating LLMs, providing valuable insights into their reliability and their propensity to introduce code smells in code generation tasks.

How Propense Are Large Language Models at Producing Code Smells? A Benchmarking Study

TL;DR

This work addresses the gap that traditional accuracy metrics fail to capture when evaluating code generation by LLMs. It introduces CodeSmellEval, consisting of the Propensity Smelly Score (PSC), the CodeSmellData dataset, and a protocol for estimating smell propensity from logits in decoder-based transformers. A case study with CodeLlama and Mistral demonstrates that both models exhibit substantial propensity to generate code smells, with 10 of 13 smells surpassing a propensity threshold of and distinct distributions across smells. The benchmark provides an interpretable, model-agnostic framework and a curated data resource to improve evaluation of code quality beyond accuracy, guiding future mitigation and research into the causes and remedies of smell generation in code produced by LLMs.

Abstract

Large Language Models (LLMs) have shown significant potential in automating software engineering tasks, particularly in code generation. However, current evaluation benchmarks, which primarily focus on accuracy, fall short in assessing the quality of the code generated by these models, specifically their tendency to produce code smells. To address this limitation, we introduce CodeSmellEval, a benchmark designed to evaluate the propensity of LLMs for generating code smells. Our benchmark includes a novel metric: Propensity Smelly Score (PSC), and a curated dataset of method-level code smells: CodeSmellData. To demonstrate the use of CodeSmellEval, we conducted a case study with two state-of-the-art LLMs, CodeLlama and Mistral. The results reveal that both models tend to generate code smells, such as simplifiable-condition and consider-merging-isinstance. These findings highlight the effectiveness of our benchmark in evaluating LLMs, providing valuable insights into their reliability and their propensity to introduce code smells in code generation tasks.

Paper Structure

This paper contains 9 sections, 2 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Benchmark
  3. Propensity Smelly Score (PSC)
  4. CodeSmellData
  5. Protocol
  6. Case Study
  7. Results & Discussion
  8. Related Work
  9. Future Plans

Figures (3)

  • Figure 1: Bootstrapped propensity of smells (100 instances per sample). The red line at $0.5$ indicates the error threshold.
  • Figure 2: Example of PSC computation for invalid-name (C0103) and comparison-of-constants (R0133), with the former surpassing the propensity threshold $\lambda = 0.5$.
  • Figure 3: Comparison of edge propensity distributions between (consider-merging-isinstance - R1701) and (disallowed-name - C2401) for both M1 and M2.