DICE: Detecting In-distribution Contamination in LLM's Fine-tuning Phase for Math Reasoning

TL;DR

It is argued that in-distribution contamination can lead to the performance drop on OOD benchmarks, and proposed DICE, a novel method that leverages the internal states of LLMs to locate-then-detect the contamination, is proposed.

Abstract

The advancement of large language models (LLMs) relies on evaluation using public benchmarks, but data contamination can lead to overestimated performance. Previous researches focus on detecting contamination by determining whether the model has seen the exact same data during training. Besides, prior work has already shown that even training on data similar to benchmark data inflates performance, namely \emph{In-distribution contamination}. In this work, we argue that in-distribution contamination can lead to the performance drop on OOD benchmarks. To effectively detect in-distribution contamination, we propose DICE, a novel method that leverages the internal states of LLMs to locate-then-detect the contamination. DICE first identifies the most sensitive layer to contamination, then trains a classifier based on the internal states of that layer. Experiments reveal DICE's high accuracy in detecting in-distribution contamination across various LLMs and math reasoning datasets. We also show the generalization capability of the trained DICE detector, which is able to detect contamination across multiple benchmarks with similar distributions. Additionally, we find that DICE's predictions correlate with the performance of LLMs fine-tuned by either us or other organizations, achieving a coefficient of determination ($R^2$) between 0.61 and 0.75. The code and data are available at https://github.com/THU-KEG/DICE.

Figures (7)

• Figure 1: Traditional contamination detection methods cannot handle in-distribution contamination. Vanilla LLM refers to the LLM fine-tuned with uncontaminated data.
• Figure 2: Overview of DICE. Locate-then-Detect LLM's in-distribution data contamination.
• Figure 3: The t-SNE results of datasets, where MATH and TabMWP are OOD and GSM-hard is ID for GSM8K.
• Figure 4: (a) Analyses on the contaminated layer location of LLaMA2-7B trained on GSM-i, which is $\theta_\text{contaminated}$. We set the LLaMA2-7B trained on OpenOrca as $\theta_\text{uncontaminated}$. (b) We change the contamination dataset as MATH and fine-tune a LLaMA2-7B on MATH as the $\theta_\text{contaminated}$. (c) We change base model as Phi2-2.7B and fine-tune two models on OpenOrca and GSM-i for analysis.
• Figure 5: (a) We perform in-distribution detection on GSM-ii with DICE trained on different layers' features. For testing, $\theta_\text{contaminated}$ is LLaMA2-7B trained on GSM-ii, and $\theta_\text{uncontaminated}$ is the LLaMA2-7B-base. (b) We change the contamination dataset as MATH. (c) We change base model as Phi2.