Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarks

TL;DR

The paper addresses whether localization methods truly pinpoint memorized data within LLMs by introducing two complementary benchmarks: INJ, which injects ground-truth memorized weights to directly assess localization accuracy, and DEL, which uses dropout to evaluate the causal role of predicted neurons for pretrained memorized sequences. Five localization methods are compared—Zero-Out, Activations, Integrated Gradients (IG), Slimming, and Hard Concrete—with pruning-based approaches (Slimming and Hard Concrete) achieving the strongest localization signals across both benchmarks. Results show all methods can localize better than random, but precise, sequence-specific localization remains challenging due to memory sharing across related sequences. The two benchmarks yield consistent method rankings and reveal that memorized information is distributed across layers, suggesting limitations for perfect, per-sequence localization but clear potential for targeted forgetting or unlearning applications. Overall, the study provides a rigorous framework to evaluate localization in LLMs and highlights the promise and current limits of existing methods for identifying and potentially removing memorized content.

Abstract

The concept of localization in LLMs is often mentioned in prior work; however, methods for localization have never been systematically and directly evaluated. We propose two complementary benchmarks that evaluate the ability of localization methods to pinpoint LLM components responsible for memorized data. In our INJ benchmark, we actively inject a piece of new information into a small subset of LLM weights, enabling us to directly evaluate whether localization methods can identify these "ground truth" weights. In our DEL benchmark, we evaluate localization by measuring how much dropping out identified neurons deletes a memorized pretrained sequence. Despite their different perspectives, our two benchmarks yield consistent rankings of five localization methods. Methods adapted from network pruning perform well on both benchmarks, and all evaluated methods show promising localization ability. On the other hand, even successful methods identify neurons that are not specific to a single memorized sequence.

Figures (6)

• Figure 1: Left:INJ Benchmark updates a small set of LLM weights to store the new piece of data, where the fine-tuned weight vectors and the corresponding neurons are filled with blue. The neurons predicted by a localization method are circled with black. $\,$ denotes true-positive, $\,$ false-positive, and $\,$ false-negative neurons. Right:DEL Benchmark drops out the predicted neurons $\,$ on a memorized pretrained sequence. A large change in Levenshtein distance after dropout indicates that $\,$ were important for LLM $f$ to retrieve the memorized sequence.
• Figure 2: The confusion matrix of Hard Concrete on a subset of data memorized by GPT2-XL.
• Figure 3: Dropout in one layer vs. multiple layers.
• Figure 4: The DEL Benchmark results of Zero-Out, IG, and Activations methods when dropping out the same number of neurons in a single layer, where the blue lines show $\Delta$ Self-Acc and the red lines show $\Delta$ Neg-Acc. Under the same "neuron budget’’, dropping out neurons in multiple layers (blue dashed lines) substantially outperforms dropout in a single layer, implying that memorized information is stored in a distributed fashion over multiple layers. Besides, dropping out neurons in the bottom layer greatly hurts the memorization of negative examples (red lines), suggesting that the bottom layer encodes general information.
• Figure 5: Confusion matrices of localization methods on a subset of sequences memorized by GPT2-XL, where each row/column represents a sequence. Different methods show similar patterns of confusion.