Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Code Copycat Conundrum: Demystifying Repetition in LLM-based Code Generation

Mingwei Liu, Juntao Li, Ying Wang, Xueying Du, Zuoyu Ou, Qiuyuan Chen, Bingxu An, Zhao Wei, Yong Xu, Fangming Zou, Xin Peng, Yiling Lou

TL;DR

The paper addresses the problem of repetition in LLM-generated code, demonstrating that repetition is pervasive across 19 code LLMs and multiple benchmarks. It conducts a two-pronged study: a quantitative analysis of repetition using three metrics across model types and tasks, and a qualitative taxonomy of 20 repetition patterns. The authors introduce DeRep, a rule-based mitigation consisting of detection and repair phases, and show that it substantially reduces repetition and improves Pass@1, with strong performance in industrial deployments. The work highlights practical implications for code generation tools, emphasizing data quality and alignment, and provides a replication package to enable broader evaluation and adoption.

Abstract

Despite recent advances in Large Language Models (LLMs) for code generation, the quality of LLM-generated code still faces significant challenges. One significant issue is code repetition, which refers to the model's tendency to generate structurally redundant code, resulting in inefficiencies and reduced readability. To address this, we conduct the first empirical study to investigate the prevalence and nature of repetition across 19 state-of-the-art code LLMs using three widely-used benchmarks. Our study includes both quantitative and qualitative analyses, revealing that repetition is pervasive and manifests at various granularities and extents, including character, statement, and block levels. We further summarize a taxonomy of 20 repetition patterns. Building on our findings, we propose DeRep, a rule-based technique designed to detect and mitigate repetition in generated code. We evaluate DeRep using both open-source benchmarks and in an industrial setting. Our results demonstrate that DeRep significantly outperforms baselines in reducing repetition (with an average improvements of 91.3%, 93.5%, and 79.9% in rep-3, rep-line, and sim-line metrics) and enhancing code quality (with a Pass@1 increase of 208.3% over greedy search). Furthermore, integrating DeRep improves the performance of existing repetition mitigation methods, with Pass@1 improvements ranging from 53.7% to 215.7%.

Code Copycat Conundrum: Demystifying Repetition in LLM-based Code Generation

TL;DR

The paper addresses the problem of repetition in LLM-generated code, demonstrating that repetition is pervasive across 19 code LLMs and multiple benchmarks. It conducts a two-pronged study: a quantitative analysis of repetition using three metrics across model types and tasks, and a qualitative taxonomy of 20 repetition patterns. The authors introduce DeRep, a rule-based mitigation consisting of detection and repair phases, and show that it substantially reduces repetition and improves Pass@1, with strong performance in industrial deployments. The work highlights practical implications for code generation tools, emphasizing data quality and alignment, and provides a replication package to enable broader evaluation and adoption.

Abstract

Despite recent advances in Large Language Models (LLMs) for code generation, the quality of LLM-generated code still faces significant challenges. One significant issue is code repetition, which refers to the model's tendency to generate structurally redundant code, resulting in inefficiencies and reduced readability. To address this, we conduct the first empirical study to investigate the prevalence and nature of repetition across 19 state-of-the-art code LLMs using three widely-used benchmarks. Our study includes both quantitative and qualitative analyses, revealing that repetition is pervasive and manifests at various granularities and extents, including character, statement, and block levels. We further summarize a taxonomy of 20 repetition patterns. Building on our findings, we propose DeRep, a rule-based technique designed to detect and mitigate repetition in generated code. We evaluate DeRep using both open-source benchmarks and in an industrial setting. Our results demonstrate that DeRep significantly outperforms baselines in reducing repetition (with an average improvements of 91.3%, 93.5%, and 79.9% in rep-3, rep-line, and sim-line metrics) and enhancing code quality (with a Pass@1 increase of 208.3% over greedy search). Furthermore, integrating DeRep improves the performance of existing repetition mitigation methods, with Pass@1 improvements ranging from 53.7% to 215.7%.

Paper Structure

This paper contains 25 sections, 3 equations, 9 figures, 7 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Empirical Study Setup
  4. Studied Code LLMs
  5. Dataset
  6. Metrics
  7. Experimental Procedure
  8. Empirical Results
  9. RQ1 Results
  10. RQ1.a (Repetition via different Metrics)
  11. RQ1.b (Repetition across Different Models)
  12. RQ1.c (Input/Output Impact on repetition)
  13. RQ2 Results (Repetition Patterns)
  14. Rule-based Repetition Mitigation
  15. Repetition Detection
  16. ...and 10 more sections

Figures (9)

  • Figure 1: Examples of Endless Repetition Cases of Code LLMs
  • Figure 2: rep_n across Different n-grams on H-P+MBPP
  • Figure 3: Distribution of rep-3, rep-line, and sim-line Metrics across Models
  • Figure 4: Trends of Metrics by Input Tokens Number Intervals
  • Figure 5: Trends of Repetition Metrics by Ground Truth Tokens Number
  • ...and 4 more figures