Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Data Contamination Detection for Modern Large Language Models: Limitations, Inconsistencies, and Oracle Challenges

Vinay Samuel, Yue Zhou, Henry Peng Zou

TL;DR

The paper tackles the problem of data contamination in large language model evaluations by empirically evaluating five detection methods across eight challenging benchmarks using four SOTA LLMs and an oracle setup that simulates instruction-finetuning contamination. It reveals substantial limitations in current detection approaches, particularly their inconsistent signals and inability to reliably detect fine-tuning contamination. The study also shows minimal agreement among methods on modern benchmarks, calling into question the reliability of existing contamination-evaluation practices. The work highlights the need for robust, generalizable contamination-evaluation frameworks to ensure trustworthy benchmarking of LLM capabilities.

Abstract

As large language models achieve increasingly impressive results, questions arise about whether such performance is from generalizability or mere data memorization. Thus, numerous data contamination detection methods have been proposed. However, these approaches are often validated with traditional benchmarks and early-stage LLMs, leaving uncertainty about their effectiveness when evaluating state-of-the-art LLMs on the contamination of more challenging benchmarks. To address this gap and provide a dual investigation of SOTA LLM contamination status and detection method robustness, we evaluate five contamination detection approaches with four state-of-the-art LLMs across eight challenging datasets often used in modern LLM evaluation. Our analysis reveals that (1) Current methods have non-trivial limitations in their assumptions and practical applications; (2) Notable difficulties exist in detecting contamination introduced during instruction fine-tuning with answer augmentation; and (3) Limited consistencies between SOTA contamination detection techniques. These findings highlight the complexity of contamination detection in advanced LLMs and the urgent need for further research on robust and generalizable contamination evaluation. Our code is available at https://github.com/vsamuel2003/data-contamination.

Towards Data Contamination Detection for Modern Large Language Models: Limitations, Inconsistencies, and Oracle Challenges

TL;DR

The paper tackles the problem of data contamination in large language model evaluations by empirically evaluating five detection methods across eight challenging benchmarks using four SOTA LLMs and an oracle setup that simulates instruction-finetuning contamination. It reveals substantial limitations in current detection approaches, particularly their inconsistent signals and inability to reliably detect fine-tuning contamination. The study also shows minimal agreement among methods on modern benchmarks, calling into question the reliability of existing contamination-evaluation practices. The work highlights the need for robust, generalizable contamination-evaluation frameworks to ensure trustworthy benchmarking of LLM capabilities.

Abstract

As large language models achieve increasingly impressive results, questions arise about whether such performance is from generalizability or mere data memorization. Thus, numerous data contamination detection methods have been proposed. However, these approaches are often validated with traditional benchmarks and early-stage LLMs, leaving uncertainty about their effectiveness when evaluating state-of-the-art LLMs on the contamination of more challenging benchmarks. To address this gap and provide a dual investigation of SOTA LLM contamination status and detection method robustness, we evaluate five contamination detection approaches with four state-of-the-art LLMs across eight challenging datasets often used in modern LLM evaluation. Our analysis reveals that (1) Current methods have non-trivial limitations in their assumptions and practical applications; (2) Notable difficulties exist in detecting contamination introduced during instruction fine-tuning with answer augmentation; and (3) Limited consistencies between SOTA contamination detection techniques. These findings highlight the complexity of contamination detection in advanced LLMs and the urgent need for further research on robust and generalizable contamination evaluation. Our code is available at https://github.com/vsamuel2003/data-contamination.
Paper Structure (17 sections, 1 figure, 4 tables)

This paper contains 17 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmark Datasets
  4. Evaluated Methods and Limitations
  5. Experiments
  6. Implementation Details
  7. Min-K% Prob
  8. Canonical Order Statistical Testing
  9. Token Completion Overlap Score
  10. Word Perturbation Quiz
  11. Local Order Quiz (Ours)
  12. Oracle Setup
  13. Main Results
  14. Conclusions and Future Work
  15. Prompts
  16. ...and 2 more sections

Figures (1)

  • Figure 1: Overview of five methods evaluated for detecting data contamination in large language models (LLMs). (a) Token Completion Overlap Score: Evaluates LLM contamination by prompting the model with partial text to see if the completion matches a reference instance. (b) Min-K% Probability: Determines potential contamination by assessing the average log-likelihood of the k% least probable tokens in a text. (c) Word Perturbation Quiz: Detects contamination by checking if the LLM can distinguish between a word-level perturbed instance and the original. (d) Instance Local Order Quiz: A prompt-based method we developed, assessing whether the LLM can identify the correct subsequent example in a dataset sequence. (e) Canonical Order Statistical Testing: Tests if the LLM shows a preference for canonical order over random shuffling, indicating potential contamination.