PDR: A Plug-and-Play Positional Decay Framework for LLM Pre-training Data Detection

TL;DR

This work identifies a critical limitation of existing likelihood-based pre-training data detection methods: uniform token-score aggregation dilutes memorization signals that are concentrated in the initial, high-entropy portion of a sequence. It introduces Positional Decay Reweighting (PDR), a training-free, plug-and-play framework that applies monotone decay weights over token positions to emphasize early signals when computing detection scores. Through extensive experiments on WikiMIA and MIMIR across multiple backbones and detection baselines (e.g., Loss, Ref, Min-$k$%++, FSD), PDR yields consistent AUROC gains, with more pronounced improvements on longer sequences and challenging settings. The method is demonstrated to be robust via ablations and bootstrap significance testing, and is positioned as a practical tool for auditing data memorization in LLMs, potentially informing defenses like differential privacy or unlearning. In summary, PDR provides a principled, versatile prior that enhances current detection techniques without retraining models, advancing our ability to audit and mitigate memorization risks in large language models.

Abstract

Detecting pre-training data in Large Language Models (LLMs) is crucial for auditing data privacy and copyright compliance, yet it remains challenging in black-box, zero-shot settings where computational resources and training data are scarce. While existing likelihood-based methods have shown promise, they typically aggregate token-level scores using uniform weights, thereby neglecting the inherent information-theoretic dynamics of autoregressive generation. In this paper, we hypothesize and empirically validate that memorization signals are heavily skewed towards the high-entropy initial tokens, where model uncertainty is highest, and decay as context accumulates. To leverage this linguistic property, we introduce Positional Decay Reweighting (PDR), a training-free and plug-and-play framework. PDR explicitly reweights token-level scores to amplify distinct signals from early positions while suppressing noise from later ones. Extensive experiments show that PDR acts as a robust prior and can usually enhance a wide range of advanced methods across multiple benchmarks.

Figures (16)

• Figure 1: Visualization of (a) token-level entropy on subsets of the challenging Mimir dataset and (b) the average token-level log-probability for members and non-members on WikiMIA dataset for LLaMA-13B model.
• Figure 2: Overview of Positional Decay Reweighting (PDR). Our method reweights the predictive probabilities of input samples based on token positions, emphasizing early tokens with higher weights. This reweighting enhances the distinction between member and non-member samples by amplifying critical signals in the score $\mathcal{S}$, making it more effective for MIA. The framework is training-free, also plug-and-play for likelihood-based scoring methods.
• Figure 2: AUROC scores of various MIA methods over five Pythia models on the Mimir dataset. Pub and Wiki denote Pubmed Central and Wikipedia (en). Avg* scores are computed by excluding Arxiv and HackerNews. $^\dag$Neighbor results are from mia_mink_plus, induces significant extra computational cost than others ($25\times$ in this case), for which reason we don't run on the 12B model.
• Figure 3: AUROC comparison of our LPDR method when integrated with Min-$k$% and Min-$k$%++ across various LLMs on WikiMIA dataset within the FSD framework.
• Figure 4: AUC comparison of different $\mathbf{\alpha}$ for LPDR on log-likelihood methods (Loss, Ref, Min-$k$%, Min-$k$%++), results from Pythia-12B model at WikiMIA ori. of 64 length.