Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Adverse Effects of Code Duplication in Machine Learning Models of Code

Miltiadis Allamanis

TL;DR

This paper reveals that code duplication in large code corpora biases machine learning evaluations, often inflating performance metrics. It develops a near-duplicate detection approach and duplication index to quantify cross-set duplication across ten datasets, and demonstrates that deduplication can substantially alter results, especially for identifier-focused tasks and higher-capacity models. The authors provide a principled framework, practical tools, and best-practice guidelines to mitigate duplication, aiming to make ML-on-code research more reliable and applicable to real-world software engineering. By highlighting the distinction between true data distribution and duplicated data, the work calls for careful evaluation design and paves the way for more robust, duplication-tolerant models and benchmarks.

Abstract

The field of big code relies on mining large corpora of code to perform some learning task. A significant threat to this approach has been recently identified by Lopes et al. (2017) who found a large amount of near-duplicate code on GitHub. However, the impact of code duplication has not been noticed by researchers devising machine learning models for source code. In this work, we explore the effects of code duplication on machine learning models showing that reported performance metrics are sometimes inflated by up to 100% when testing on duplicated code corpora compared to the performance on de-duplicated corpora which more accurately represent how machine learning models of code are used by software engineers. We present a duplication index for widely used datasets, list best practices for collecting code corpora and evaluating machine learning models on them. Finally, we release tools to help the community avoid this problem in future research.

The Adverse Effects of Code Duplication in Machine Learning Models of Code

TL;DR

This paper reveals that code duplication in large code corpora biases machine learning evaluations, often inflating performance metrics. It develops a near-duplicate detection approach and duplication index to quantify cross-set duplication across ten datasets, and demonstrates that deduplication can substantially alter results, especially for identifier-focused tasks and higher-capacity models. The authors provide a principled framework, practical tools, and best-practice guidelines to mitigate duplication, aiming to make ML-on-code research more reliable and applicable to real-world software engineering. By highlighting the distinction between true data distribution and duplicated data, the work calls for careful evaluation design and paves the way for more robust, duplication-tolerant models and benchmarks.

Abstract

The field of big code relies on mining large corpora of code to perform some learning task. A significant threat to this approach has been recently identified by Lopes et al. (2017) who found a large amount of near-duplicate code on GitHub. However, the impact of code duplication has not been noticed by researchers devising machine learning models for source code. In this work, we explore the effects of code duplication on machine learning models showing that reported performance metrics are sometimes inflated by up to 100% when testing on duplicated code corpora compared to the performance on de-duplicated corpora which more accurately represent how machine learning models of code are used by software engineers. We present a duplication index for widely used datasets, list best practices for collecting code corpora and evaluating machine learning models on them. Finally, we release tools to help the community avoid this problem in future research.

Paper Structure

This paper contains 21 sections, 4 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Code Duplication & Machine Learning
  3. Concepts and Definitions
  4. Duplication Bias
  5. Measuring Duplication
  6. Detecting near-duplicates
  7. Duplication Statistics
  8. Human Evaluation
  9. Impact on Machine Learning Models
  10. Terminology
  11. Biased vs. Unbiased Performance
  12. Autocompletion via Language Modeling
  13. Performance
  14. Model Capacity and Impact on Code Duplication
  15. Other Models and Tasks
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Schematic description of types of duplicates. The dashed boxes indicate the subset of files that are duplicates within each set.
  • Figure 2: The impact of code duplication on the NLM with different capacity trained on JavaScript-150k. The solid lines show the accuracy of the NLM model when predicting all tokens, whereas the dashed lines show the accuracy of predicting only identifiers. Blue lines indicate the cross-set biased accuracy, and black ones show the unbiased test accuracy. The larger the capacity of the model, the more severe the impact of code duplication (red shaded area).