Concerned with Data Contamination? Assessing Countermeasures in Code Language Model
Jialun Cao, Wuqi Zhang, Shing-Chi Cheung
TL;DR
This work tackles data contamination as a threat to evaluating code language models by systematically comparing contaminated (pre-cutoff) code with cleansed data produced via countermeasures. Through a large-scale study using about 2 million Python functions collected from 2018–2023, the authors test three countermeasures—recent data, curated benchmarks, and code refactoring—across multiple CLMs and a code-completion task. They find that these countermeasures do not reliably mitigate contamination: CLMs can perform better on recent or curated data, and even refactoring can boost performance; moreover, existing membership-inference metrics struggle to distinguish contaminated from cleansed data. The study highlights the growing influence of AI-assisted code in training data and benchmarks, and it argues for more robust, quantitative evaluation strategies to accurately assess CLM capabilities for software engineering tasks.
Abstract
Various techniques have been proposed to leverage the capabilities of code language models (CLMs) for SE tasks. While these techniques typically evaluate their effectiveness using publicly available datasets, the evaluation can be subject to data contamination threats where the evaluation datasets have already been used to train the concerned CLMs. This can significantly affect the reliability of the evaluation. Different countermeasures have been suggested to mitigate the data contamination threat. Countermeasures include using more recent data, curating new data, and refactoring existing data are introduced, yet it is unclear whether these countermeasures could really mitigate data contamination threats to model evaluation. To fill the gap, we systematically study to quantify the impacts of these countermeasures on CLMs' performance. To facilitate the study, we collected over 2 million Python functions with timestamps ranging from January 1st, 2018, to December 31st, 2023. The data created before the models' cut-off date are considered "contaminated data", while the data where the countermeasures are taken are regarded as "cleansed data". We study the impact of these countermeasures by investigating the difference in CLMs' performance on contaminated and cleansed data derived from different countermeasures. Our experiments yield several interesting observations. For instance, CLMs do not necessarily perform worse on data after the models' cut-off date; on the contrary, they sometimes perform better. In addition, refactoring did not always result in decreased performance; it could lead to improvements instead. Furthermore, existing metrics such as perplexity cannot distinguish contaminated/cleansed data. We hope that the results and observations could help deepen the understanding of CLMs' capabilities and inform the community about data contamination.