Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RN-F: A Novel Approach for Mitigating Contaminated Data in Large Language Models

Le Vu Anh, Dinh Duc Nha Nguyen, Phi Long Nguyen

TL;DR

Data contamination in LLM benchmarks challenges the reliability of evaluations. RN-F introduces Residual-Noise Fingerprinting, a gradient-free, single-pass detector that exploits per-layer quantization residuals $r_\ell(x)$ from post-training 4-bit quantization to distinguish clean from tainted inputs, with $r_{\max}(x)=\max_\ell r_\ell(x)$ aggregating signals across layers. The approach comes with a theoretical analysis showing sub-Gaussian tails for clean data and bounded mean shifts for contaminated data, plus a calibration procedure using a small clean dataset to set a robust threshold. Empirically, RN-F outperforms state-of-the-art detectors on multiple compact models and modalities (image/text/tabular) with modest overhead (roughly 3.5–4.3% latency and 3.6–4.4% energy) and demonstrates strong cross-modal detectability, indicating practical utility for edge and TinyML deployments.

Abstract

Large Language Models (LLMs) have become foundational in modern artificial intelligence, powering a wide range of applications from code generation and virtual assistants to scientific research and enterprise automation. However, concerns about data contamination--where test data overlaps with training data--have raised serious questions about the reliability of these applications. Despite awareness of this issue, existing methods fall short in effectively identifying or mitigating contamination. In this paper, we propose Residual-Noise Fingerprinting (RN-F), a novel framework for detecting contaminated data in LLMs. RN-F is a single-pass, gradient-free detection method that leverages residual signal patterns without introducing additional floating-point operations. Our approach is lightweight, model-agnostic, and efficient. We evaluate RN-F on multiple LLMs across various contaminated datasets and show that it consistently outperforms existing state-of-the-art methods, achieving performance improvements of up to 10.5% in contamination detection metrics.

RN-F: A Novel Approach for Mitigating Contaminated Data in Large Language Models

TL;DR

Data contamination in LLM benchmarks challenges the reliability of evaluations. RN-F introduces Residual-Noise Fingerprinting, a gradient-free, single-pass detector that exploits per-layer quantization residuals from post-training 4-bit quantization to distinguish clean from tainted inputs, with aggregating signals across layers. The approach comes with a theoretical analysis showing sub-Gaussian tails for clean data and bounded mean shifts for contaminated data, plus a calibration procedure using a small clean dataset to set a robust threshold. Empirically, RN-F outperforms state-of-the-art detectors on multiple compact models and modalities (image/text/tabular) with modest overhead (roughly 3.5–4.3% latency and 3.6–4.4% energy) and demonstrates strong cross-modal detectability, indicating practical utility for edge and TinyML deployments.

Abstract

Large Language Models (LLMs) have become foundational in modern artificial intelligence, powering a wide range of applications from code generation and virtual assistants to scientific research and enterprise automation. However, concerns about data contamination--where test data overlaps with training data--have raised serious questions about the reliability of these applications. Despite awareness of this issue, existing methods fall short in effectively identifying or mitigating contamination. In this paper, we propose Residual-Noise Fingerprinting (RN-F), a novel framework for detecting contaminated data in LLMs. RN-F is a single-pass, gradient-free detection method that leverages residual signal patterns without introducing additional floating-point operations. Our approach is lightweight, model-agnostic, and efficient. We evaluate RN-F on multiple LLMs across various contaminated datasets and show that it consistently outperforms existing state-of-the-art methods, achieving performance improvements of up to 10.5% in contamination detection metrics.
Paper Structure (29 sections, 2 theorems, 13 equations, 4 figures, 3 tables, 1 algorithm)

This paper contains 29 sections, 2 theorems, 13 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Problem set-up
  5. Why does the residual separate clean from tainted inputs?
  6. Uniform quantization as structured noise.
  7. Contaminated inputs break the symmetry.
  8. Distributional analysis
  9. Proof sketch.
  10. Corollary (layer max).
  11. Practical calibration
  12. Connection to Fisher information.
  13. Proposed Framework
  14. Complexity.
  15. Experiments and Evaluations
  16. ...and 14 more sections

Key Result

Proposition 3.1

Assume each coordinate rounding error $\varepsilon\sim\mathrm{Unif}[-q,q]$ and that the layer mapping $x\mapsto h_\ell(x)$ is $K$-Lipschitz. Then for any ID input $x$ and any $\tau>0$, where $\mu_\ell=\mathbb{E}[r_\ell]$.

Figures (4)

  • Figure 1: Layer-1 quantization residuals reveal anomalous inputs. Clean data cluster near zero (blue), whereas memorised (orange) and back-door (red) samples shift the distribution to the right.
  • Figure 2: RN-F calibration curves on TabPFNGen. Performance saturates well before the 512-example buffer used in Section \ref{['sec:maindetector']}.
  • Figure 3: Instance-level performance comparison across three workloads from the M5Product benchmark. RN-F consistently outperforms contamination detectors CDD, BAIT, and ConStat across Accuracy, macro-F$_1$, and ROC-AUC.
  • Figure 4: Correlation of quantisation–residual spikes between image, text and tabular branches on the M5Product benchmark. The strong off-diagonal values indicate that contamination affects modalities in a coordinated manner, which RN-F exploits by pooling residuals.

Theorems & Definitions (3)

  • Definition 3.1: Layer-wise residual
  • Proposition 3.1: Sub-Gaussian tail on ID data
  • Theorem 3.2: Instance-level detection guarantee