Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding Verbatim Memorization in LLMs Through Circuit Discovery

Ilya Lasy, Peter Knees, Stefan Woltran

TL;DR

This work addresses verbatim memorization in large language models by identifying minimal circuits that trigger and sustain memorized content. It develops two contrastive datasets on the Wikipedia subset of The Pile and applies Edge Attribution Patching with Integrated Gradients to extract compact, faithful circuits governing memorization. The authors find distinct circuits for initiation and maintenance, with trigger circuits capable of starting memorization and maintaining it, while maintenance-only circuits cannot initiate it. Cross-task and cross-corpus analysis shows memorization-prevention circuits generalize more robustly than memorization-induction circuits, offering practical implications for safer and more controllable LLM behavior.

Abstract

Underlying mechanisms of memorization in LLMs -- the verbatim reproduction of training data -- remain poorly understood. What exact part of the network decides to retrieve a token that we would consider as start of memorization sequence? How exactly is the models' behaviour different when producing memorized sentence vs non-memorized? In this work we approach these questions from mechanistic interpretability standpoint by utilizing transformer circuits -- the minimal computational subgraphs that perform specific functions within the model. Through carefully constructed contrastive datasets, we identify points where model generation diverges from memorized content and isolate the specific circuits responsible for two distinct aspects of memorization. We find that circuits that initiate memorization can also maintain it once started, while circuits that only maintain memorization cannot trigger its initiation. Intriguingly, memorization prevention mechanisms transfer robustly across different text domains, while memorization induction appears more context-dependent.

Understanding Verbatim Memorization in LLMs Through Circuit Discovery

TL;DR

This work addresses verbatim memorization in large language models by identifying minimal circuits that trigger and sustain memorized content. It develops two contrastive datasets on the Wikipedia subset of The Pile and applies Edge Attribution Patching with Integrated Gradients to extract compact, faithful circuits governing memorization. The authors find distinct circuits for initiation and maintenance, with trigger circuits capable of starting memorization and maintaining it, while maintenance-only circuits cannot initiate it. Cross-task and cross-corpus analysis shows memorization-prevention circuits generalize more robustly than memorization-induction circuits, offering practical implications for safer and more controllable LLM behavior.

Abstract

Underlying mechanisms of memorization in LLMs -- the verbatim reproduction of training data -- remain poorly understood. What exact part of the network decides to retrieve a token that we would consider as start of memorization sequence? How exactly is the models' behaviour different when producing memorized sentence vs non-memorized? In this work we approach these questions from mechanistic interpretability standpoint by utilizing transformer circuits -- the minimal computational subgraphs that perform specific functions within the model. Through carefully constructed contrastive datasets, we identify points where model generation diverges from memorized content and isolate the specific circuits responsible for two distinct aspects of memorization. We find that circuits that initiate memorization can also maintain it once started, while circuits that only maintain memorization cannot trigger its initiation. Intriguingly, memorization prevention mechanisms transfer robustly across different text domains, while memorization induction appears more context-dependent.

Paper Structure

This paper contains 22 sections, 3 equations, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Memorization in Language Models
  4. Mechanistic Interpretability
  5. Methodology
  6. Divergence points
  7. Memorization Decision Dataset
  8. Branch Comparison Dataset
  9. Circuit Discovery
  10. Experiments
  11. Memorization Decision Task
  12. Branch Comparison Task
  13. Circuit Verification
  14. Results
  15. Cross-Task Generalization
  16. ...and 7 more sections

Figures (1)

  • Figure 1: Examples of divergence points in memorized content. Each pair shows a shortened context (left) followed by two possible continuations: the ground truth (red, top) and model's actual generation (blue, bottom) with their respective BLEU scores. Higher BLEU scores (closer to 1.0) indicate stronger memorization.