Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions

Yujuan Fu, Ozlem Uzuner, Meliha Yetisgen, Fei Xia

Abstract

Large language models (LLMs) have demonstrated great performance across various benchmarks, showing potential as general-purpose task solvers. However, as LLMs are typically trained on vast amounts of data, a significant concern in their evaluation is data contamination, where overlap between training data and evaluation datasets inflates performance assessments. Multiple approaches have been developed to identify data contamination. These approaches rely on specific assumptions that may not hold universally across different settings. To bridge this gap, we systematically review 50 papers on data contamination detection, categorize the underlying assumptions, and assess whether they have been rigorously validated. We identify and analyze eight categories of assumptions and test three of them as case studies. Our case studies focus on detecting direct, instance-level data contamination, which is also referred to as Membership Inference Attacks (MIA). Our analysis reveals that MIA approaches based on these three assumptions can have similar performance to random guessing, on datasets used in LLM pretraining, suggesting that current LLMs might learn data distributions rather than memorizing individual instances. Meanwhile, MIA can easily fail when there are data distribution shifts between the seen and unseen instances.

Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions

Abstract

Large language models (LLMs) have demonstrated great performance across various benchmarks, showing potential as general-purpose task solvers. However, as LLMs are typically trained on vast amounts of data, a significant concern in their evaluation is data contamination, where overlap between training data and evaluation datasets inflates performance assessments. Multiple approaches have been developed to identify data contamination. These approaches rely on specific assumptions that may not hold universally across different settings. To bridge this gap, we systematically review 50 papers on data contamination detection, categorize the underlying assumptions, and assess whether they have been rigorously validated. We identify and analyze eight categories of assumptions and test three of them as case studies. Our case studies focus on detecting direct, instance-level data contamination, which is also referred to as Membership Inference Attacks (MIA). Our analysis reveals that MIA approaches based on these three assumptions can have similar performance to random guessing, on datasets used in LLM pretraining, suggesting that current LLMs might learn data distributions rather than memorizing individual instances. Meanwhile, MIA can easily fail when there are data distribution shifts between the seen and unseen instances.

Paper Structure

This paper contains 41 sections, 11 equations, 20 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Literature Evaluation
  3. Levels of Data Contamination
  4. Instance-Level Contamination
  5. Dataset-Level Contamination
  6. Detection of Direct Data Contamination
  7. Instance Similarity
  8. Probability Analysis
  9. Absolute Probability
  10. Reference Probability by An Instance
  11. Reference Probability by Another LM
  12. Instance Generation and Instance Selection
  13. Verbatim Memorization
  14. Key Information Generation
  15. Generation Variation
  16. ...and 26 more sections

Figures (20)

  • Figure 1: Average MIA AUC for the Pythia-6.9b model with PPL_200, when the seen and unseen instances are from different domains. The abbreviations represent the domains in Table \ref{['tab:results_pythia-6.9b']} in the Appendix \ref{['sec:appendix_case_study']}.
  • Figure 2: The density plot of PPL_200 from the Pythia-6.9b model, when both seen and unseen instances are from the Github domain.
  • Figure 3: The density plot of PPL_200 from the Pythia-6.9b model, when both seen and unseen instances are from the Pile-CC domain.
  • Figure 4: The density plot of PPL_200 from the Pythia-9.6b model, when the seen instances are from the Github domain, and the unseen instances are from the Pile-CC domain.
  • Figure 5: The density plot of PPL_200 from the Pythia-9.6b model, when the seen instances are from the Pile-CC domain, and the unseen instances are from the Github domain.
  • ...and 15 more figures