DVD: A Robust Method for Detecting Variant Contamination in Large Language Model Evaluation

TL;DR

Variant contamination—semantically equivalent but surface-different test items—threatens the validity of LLM evaluations. The authors propose DVD, a training-free detector that uses the variance of generation distributions under temperature sampling to identify contamination, grounded in a two-state model of generation: memory-adherence and perturbation-drift. They construct a dedicated benchmark across Omni-MATH and SuperGPQA, and show that DVD consistently outperforms perplexity-, Min-K%++, CDD, and embedding-similarity baselines across model scales and fine-tuning regimes. This approach provides a principled, robust fingerprint for fairer, more trustworthy evaluation of large language models in diverse domains.

Abstract

Evaluating large language models (LLMs) is increasingly confounded by \emph{variant contamination}: the training corpus contains semantically equivalent yet lexically or syntactically altered versions of test items. Unlike verbatim leakage, these paraphrased or structurally transformed variants evade existing detectors based on sampling consistency or perplexity, thereby inflating benchmark scores via memorization rather than genuine reasoning. We formalize this problem and introduce \textbf{DVD} (\textbf{D}etection via \textbf{V}ariance of generation \textbf{D}istribution), a single-sample detector that models the local output distribution induced by temperature sampling. Our key insight is that contaminated items trigger alternation between a \emph{memory-adherence} state and a \emph{perturbation-drift} state, yielding abnormally high variance in the synthetic difficulty of low-probability tokens; uncontaminated items remain in drift with comparatively smooth variance. We construct the first benchmark for variant contamination across two domains Omni-MATH and SuperGPQA by generating and filtering semantically equivalent variants, and simulate contamination via fine-tuning models of different scales and architectures (Qwen2.5 and Llama3.1). Across datasets and models, \textbf{DVD} consistently outperforms perplexity-based, Min-$k$\%++, edit-distance (CDD), and embedding-similarity baselines, while exhibiting strong robustness to hyperparameters. Our results establish variance of the generation distribution as a principled and practical fingerprint for detecting variant contamination in LLM evaluation.

Figures (11)

• Figure 1: Performance of traditional contamination detection methods on the variant contamination identification task. In the figure, an AUC value below 0.5 indicates that the model’s predictions are inversely correlated with the true labels, while an AUC close to 0.5 suggests performance equivalent to random guessing.
• Figure 2: Our Method Pipeline
• Figure 3: Performance comparison of the DVD method and other baselines on the Llama architecture (using the Llama3.1-8B-Instruct model). EM denotes the Embedding-similarity method. A complete visualization of the results is provided in Figure \ref{['fig:ablationStudyall']} of Appendix \ref{['sec:ablationStudyall']}.
• Figure 4: Average performance of the DVD method across different epochs.
• Figure 5: The DVD method demonstrates remarkable robustness across a wide range of hyperparameters.