Demystifying Verbatim Memorization in Large Language Models

TL;DR

The paper investigates verbatim memorization in large language models by introducing a controlled sequence-injection framework that continues pre-training from a checkpoint with injected sequences. It shows that memorization requires non-trivial repetition, is enhanced in higher-quality and larger models, and arises from distributed abstract model states that interact with general language modeling rather than isolated weights. Through a suite of causal and cross-model interventions, the authors demonstrate that memorization is intertwined with general LM capabilities and that unlearning methods struggle to remove memorized content without degrading performance. The work highlights that memorization cannot be cleanly isolated or easily suppressed, underscoring the need to study and address abstract triggering states to manage privacy and copyright risks in LLMs.

Abstract

Large Language Models (LLMs) frequently memorize long sequences verbatim, often with serious legal and privacy implications. Much prior work has studied such verbatim memorization using observational data. To complement such work, we develop a framework to study verbatim memorization in a controlled setting by continuing pre-training from Pythia checkpoints with injected sequences. We find that (1) non-trivial amounts of repetition are necessary for verbatim memorization to happen; (2) later (and presumably better) checkpoints are more likely to verbatim memorize sequences, even for out-of-distribution sequences; (3) the generation of memorized sequences is triggered by distributed model states that encode high-level features and makes important use of general language modeling capabilities. Guided by these insights, we develop stress tests to evaluate unlearning methods and find they often fail to remove the verbatim memorized information, while also degrading the LM. Overall, these findings challenge the hypothesis that verbatim memorization stems from specific model weights or mechanisms. Rather, verbatim memorization is intertwined with the LM's general capabilities and thus will be very difficult to isolate and suppress without degrading model quality.

Figures (15)

• Figure 1: An overview of our sequence injection framework, which creates a control model $\mathcal{M}^{(\varnothing)}$ and a treatment model $\mathcal{M}^{(X)}$ by continued pre-training from the same checkpoint, with a set of sequences to memorize $X$ injected into $\mathcal{M}^{(X)}$'s training data. Our framework explicitly creates a counterfactual state that allows us to study, via causal interventions, what the model would have been if it had not seen a particular sequence.
• Figure 2: Single-shot verbatim memorization length of the 2.8b and 6.9b models after 200 training steps.
• Figure 3: Pythia checkpoint vs. verbatim memorization length of the original and shuffled sequences.
• Figure 4: Causal dependencies between the trigger and verbatim memorized tokens. (a) An example of a memorized token that depends on the trigger (the yellow box). The example is the first sentence of the book Harry Potter and the Philosopher's Stone, which is in the Pile. (b) An example of a memorized token that does not depend on the same trigger. (c) The percentage of memorized tokens that causally depend on the trigger decreases by step. (d) For memorized tokens that depend on the trigger, there is on average one causal dependency even at the middle layers.
• Figure 5: Three sets of cross-model interchange interventions that allow us to measure to what extent models reuse components learned from general language modeling in verbatim memorization.