Memorization or Interpolation ? Detecting LLM Memorization through Input Perturbation Analysis

TL;DR

This work tackles the memorization problem in large language models by proposing PEARL, a black-box memorization detector grounded in the Perturbation Sensitivity Hypothesis (PSH). PEARL uses controlled input perturbations (bit-flip) and repeated prompting to quantify how performance degrades, distinguishing memorized content from genuine generalization without access to model internals or training data. Experiments on open Pythia variants and the GPT-4o showcase that memorized data exhibit higher perturbation sensitivity, with results modulated by model size, task type, and data source (e.g., Bible, HumanEval, NYT). The framework offers a practical tool for privacy, IP compliance, and trustworthy model evaluation, while also highlighting limitations such as the need for known-outside data to calibrate thresholds and task-dependent detectability. Overall, PEARL advances black-box memorization detection and informs responsible AI data usage and assessment practices.

Abstract

While Large Language Models (LLMs) achieve remarkable performance through training on massive datasets, they can exhibit concerning behaviors such as verbatim reproduction of training data rather than true generalization. This memorization phenomenon raises significant concerns about data privacy, intellectual property rights, and the reliability of model evaluations. This paper introduces PEARL, a novel approach for detecting memorization in LLMs. PEARL assesses how sensitive an LLM's performance is to input perturbations, enabling memorization detection without requiring access to the model's internals. We investigate how input perturbations affect the consistency of outputs, enabling us to distinguish between true generalization and memorization. Our findings, following extensive experiments on the Pythia open model, provide a robust framework for identifying when the model simply regurgitates learned information. Applied on the GPT 4o models, the PEARL framework not only identified cases of memorization of classic texts from the Bible or common code from HumanEval but also demonstrated that it can provide supporting evidence that some data, such as from the New York Times news articles, were likely part of the training data of a given model.

Figures (15)

• Figure 1: Illustration of memorisation and interpolation in a Completion task with the model amazon-nova-lite-v1.0 intelligence2024amazon
• Figure 2: GPT_4o text completion performance falloff for a memorized Shakespeare poem submitted to perturbations vs regular performance degradation with a recent text (not part of the training set of GPT_4o).
• Figure 3: Overview of the PEARL framework for identifying memorization in LLMs based on the PSH hypothesis
• Figure 4: Bit-flip perturbation injection in text inputs, with $k \in \mathbb{N}$
• Figure 5: Examples input X and reference output Y for a text completion task.