On the Effectiveness of Training Data Optimization for LLM-based Code Generation: An Empirical Study
Shiqi Kuang, Zhao Tian, Tao Xiao, Dong Wang, Junjie Chen
TL;DR
The paper addresses how training data optimization impacts LLM-based code generation by conducting a large-scale empirical study across three benchmarks, four LLMs, and five techniques (plus pairwise combinations). It finds data synthesis most improves functional correctness and reduces code smells, while maintainability benefits align with data refactoring, cleaning, and selection; most combinations do not consistently beat the best single technique for correctness, though they can boost overall code quality. The standout combination is data synthesis with data refactoring (Syn+Ref), which yields the strongest overall performance, and fine-grained analyses reveal that larger, complementary data modifications tend to drive better results. These findings offer practical guidance on selecting and sequencing data optimization techniques to balance correctness, reliability, and maintainability in real-world deployments.
Abstract
Large language models (LLMs) have achieved remarkable progress in code generation, largely driven by the availability of high-quality code datasets for effective training. To further improve data quality, numerous training data optimization techniques have been proposed; however, their overall effectiveness has not been systematically evaluated. To bridge this gap, we conduct the first large-scale empirical study, examining five widely-used training data optimization techniques and their pairwise combinations for LLM-based code generation across three benchmarks and four LLMs. Our results show that data synthesis is the most effective technique for improving functional correctness and reducing code smells, although it performs relatively worse on code maintainability compared to data refactoring, cleaning, and selection. Regarding combinations, we find that most combinations do not further improve functional correctness but can effectively enhance code quality (code smells and maintainability). Among all combinations, data synthesis combined with data refactoring achieves the strongest overall performance. Furthermore, our fine-grained analysis reinforces these findings and provides deeper insights into how individual techniques and their combinations influence code generation effectiveness. Overall, this work represents a first step toward a systematic understanding of training data optimization and combination strategies, offering practical guidance for future research and deployment in LLM-based code generation.