Rethinking Repetition Problems of LLMs in Code Generation

TL;DR

This work tackles the dominant issue of structural repetition in code generation by LLMs. It introduces RPG, a grammar-based decoding strategy that uses a pushdown automaton to reduce generated tokens to grammar rules, detects repetitions with suffix and LCP analyses, and penalizes repetitive grammar patterns during decoding. The authors also present CodeRepetEval, a dataset designed to evaluate repetition mitigation across artificial, benchmark, and real-world scenarios. Empirical results show RPG substantially reduces repetitions and improves code quality on CodeRepetEval, HumanEval, and MBPP across multiple models and programming languages, demonstrating strong generalization and practical impact for robust code generation.

Abstract

With the advent of neural language models, the performance of code generation has been significantly boosted. However, the problem of repetitions during the generation process continues to linger. Previous work has primarily focused on content repetition, which is merely a fraction of the broader repetition problem in code generation. A more prevalent and challenging problem is structural repetition. In structural repetition, the repeated code appears in various patterns but possesses a fixed structure, which can be inherently reflected in grammar. In this paper, we formally define structural repetition and propose an efficient decoding approach called RPG, which stands for Repetition Penalization based on Grammar, to alleviate the repetition problems in code generation for LLMs. Specifically, RPG first leverages grammar rules to identify repetition problems during code generation, and then strategically decays the likelihood of critical tokens that contribute to repetitions, thereby mitigating them in code generation. To facilitate this study, we construct a new dataset CodeRepetEval to comprehensively evaluate approaches for mitigating the repetition problems in code generation. Extensive experimental results demonstrate that RPG substantially outperforms the best-performing baselines on CodeRepetEval dataset as well as HumanEval and MBPP benchmarks, effectively reducing repetitions and enhancing the quality of generated code.

Figures (5)

• Figure 1: Examples of repetition problems in code generation, collected from the well-trained LLMs, e.g., CodeLlama codellama and ChatGPT ChatGPT (Left). The statistical percentage of two repetition forms occurs in the generated code of LLMs (Right).
• Figure 2: A case of structural repetition generated by CodeLlama with temperature = 0, where the dashed-underline text is the prompt (Left). The corresponding grammar rules of structural patterns (Top). LLM's probabilities of generated tokens in each repetition (Right).
• Figure 3: The performance of RPG applied to LLMs of different sizes. This result is the average value across three scenarios on CodeRepetEval dataset.
• Figure 4: Two cases of generating structural repetition and the effect of our approach on them. LLMs succumb to endless loops of repetition. Our proposed approach can effectively break out of these loops, steering back to a normal code generation trajectory, and ultimately succeeding in producing correct code.
• Figure 5: The influence of hyper-parameters $\lambda$ on Artificial Synthesis, Code Generation Benchmarks, and Real-world Repositories scenarios of CodeRepetEval dataset, as well as HumanEval and MBPP benchmarks. We use the gray dashed line to represent the employed hyper-parameters.