Quality In, Quality Out: Investigating Training Data's Role in AI Code Generation

TL;DR

This study empirically investigates whether low-quality training data propagates into low-quality code produced by DL-based code generators. Using a large-scale Python dataset from The Stack, the authors quantify training-data quality with Semgrep and compare two fine-tuned models: one trained on full data (DSC_f) and one on a cleaned dataset (DSC_c). They show that 4.98% of training functions contain quality issues and that 5.85% of code generated by DSC_f is low-quality, which drops to 2.16% when trained on cleaned data (DSC_c), without compromising functional correctness. The results provide strong evidence that curating high-quality training data improves generated code quality at inference time, underscoring the practical value of data cleaning in code-generation pipelines and suggesting directions for broader studies across languages and models.

Abstract

Deep Learning-based code generators have seen significant advancements in recent years. Tools such as GitHub Copilot are used by thousands of developers with the main promise of a boost in productivity. However, researchers have recently questioned their impact on code quality showing, for example, that code generated by DL-based tools may be affected by security vulnerabilities. Since DL models are trained on large code corpora, one may conjecture that low-quality code they output is the result of low-quality code they have seen during training. However, there is very little empirical evidence documenting this phenomenon. Indeed, most of previous work look at the frequency with which commercial code generators recommend low-quality code without the possibility of relating this to their training set. We investigate the extent to which low-quality code instances seen during training affect the quality of the code generated at inference time. We start by fine-tuning a pre-trained DL model on a large-scale dataset being representative of those usually adopted in the training of code generators. We show that 4.98% of functions in this dataset exhibit one or more quality issues related to security, maintainability, best practices, etc. We use the fine-tuned model to generate 551k Python functions, showing that 5.85% of them are affected by at least one quality issue. We then remove from the training set the low-quality functions, and use the cleaned dataset to fine-tune a second model which has been used to generate the same 551k Python functions. We show that the model trained on the cleaned dataset exhibits similar performance in terms of functional correctness as compared to the original model while, however, generating a statistically significant lower number of low-quality functions (2.16%). Our study empirically documents the importance of high-quality training data for code generators.

Figures (3)

• Figure 1: Docstring cleaning example.
• Figure 2: Types of quality issues found in the 4.98% (219,723) low-quality functions in the fine-tuning dataset
• Figure 3: Functions generated by DSC$_f$ and DSC$_c$ for the same prompt