A Comprehensive Survey of Contamination Detection Methods in Large Language Models

TL;DR

This survey provides a comprehensive taxonomy and appraisal of contamination detection methods for large language models, distinguishing open-data and closed-data scenarios and mapping techniques to required model access and training stages. It catalogues over 50 detection approaches—from string and embedding-based overlaps toMEMORIZATION, MIAs, and confidence-based signals—while highlighting their strengths, limitations, and practical deployment considerations. The authors also discuss best practices to avoid contamination, propose open-data pre-training ecosystems, introduce new evaluation benchmarks, and outline future challenges including real-time detection, evasion tactics, and legal-ethical data governance. The work aims to standardize contamination-aware evaluation and guide researchers toward reliable, trustworthy LLM assessments with broader societal impact.

Abstract

With the rise of Large Language Models (LLMs) in recent years, abundant new opportunities are emerging, but also new challenges, among which contamination is quickly becoming critical. Business applications and fundraising in Artificial Intelligence (AI) have reached a scale at which a few percentage points gained on popular question-answering benchmarks could translate into dozens of millions of dollars, placing high pressure on model integrity. At the same time, it is becoming harder and harder to keep track of the data that LLMs have seen; if not impossible with closed-source models like GPT-4 and Claude-3 not divulging any information on the training set. As a result, contamination becomes a major issue: LLMs' performance may not be reliable anymore, as the high performance may be at least partly due to their previous exposure to the data. This limitation jeopardizes real capability improvement in the field of NLP, yet, there remains a lack of methods on how to efficiently detect contamination. In this paper, we survey all recent work on contamination detection with LLMs, analyzing their methodologies and use cases to shed light on the appropriate usage of contamination detection methods. Our work calls the NLP research community's attention into systematically taking into account contamination bias in LLM evaluation.

Figures (1)

• Figure 1: Classification of contamination detection methods reviewed in this paper.

Theorems & Definitions (3)

• Definition 1: Contamination
• Definition 2: Membership Inference Attack
• Definition 3: Model Memorization