AttenMIA: LLM Membership Inference Attack through Attention Signals

TL;DR

The paper presents AttenMIA, a white-box membership inference attack that leverages self-attention signals in transformer-based LLMs to infer training-set membership. It introduces two feature families—Transitional features tracking cross-layer/head attention consistency and Perturbation features measuring attention shifts under input changes—and trains a lightweight MLP classifier on their concatenation. Across open-weight models and benchmarks (WikiMIA, MIMIR), AttenMIA achieves state-of-the-art performance, with ROC AUC up to $0.996$ and low-FPR efficacy (e.g., TPR@1%FPR of $87.9\%$), and demonstrates strong generalization across architectures. The work also shows that integrating AttenMIA into data extraction settings yields higher memorization detection quality, while pre-training deduplication offers limited mitigation, highlighting a pressing privacy risk and the need for defenses beyond deduplication.

Abstract

Large Language Models (LLMs) are increasingly deployed to enable or improve a multitude of real-world applications. Given the large size of their training data sets, their tendency to memorize training data raises serious privacy and intellectual property concerns. A key threat is the membership inference attack (MIA), which aims to determine whether a given sample was included in the model's training set. Existing MIAs for LLMs rely primarily on output confidence scores or embedding-based features, but these signals are often brittle, leading to limited attack success. We introduce AttenMIA, a new MIA framework that exploits self-attention patterns inside the transformer model to infer membership. Attention controls the information flow within the transformer, exposing different patterns for memorization that can be used to identify members of the dataset. Our method uses information from attention heads across layers and combines them with perturbation-based divergence metrics to train an effective MIA classifier. Using extensive experiments on open-source models including LLaMA-2, Pythia, and Opt models, we show that attention-based features consistently outperform baselines, particularly under the important low-false-positive metric (e.g., achieving up to 0.996 ROC AUC & 87.9% TPR@1%FPR on the WikiMIA-32 benchmark with Llama2-13b). We show that attention signals generalize across datasets and architectures, and provide a layer- and head-level analysis of where membership leakage is most pronounced. We also show that using AttenMIA to replace other membership inference attacks in a data extraction framework results in training data extraction attacks that outperform the state of the art. Our findings reveal that attention mechanisms, originally introduced to enhance interpretability, can inadvertently amplify privacy risks in LLMs, underscoring the need for new defenses.

Figures (9)

• Figure 1: Overview of AttenMIA, our attention-based membership inference framework. The pipeline consists of two parallel feature pathways: (1) Transitional features, which quantify consistency and drift of attention patterns across layers and heads (e.g., correlation, Frobenius distance, KL divergence, barycenter drift), and (2) Perturbation features, which capture how attention distributions shift under controlled input modifications (token dropping, replacement, prefix insertion). Both feature families are extracted from the target LLM and concatenated into a feature vector $\mathbf{z}$, which is used to train an MIA classifier. At evaluation time, the classifier outputs a membership probability to decide whether a target sample was part of the training data.
• Figure 2: Comparison of per-layer attention activation and perturbation effects in OPT-6.7B.
• Figure 3: Kernel density estimates of transitional attention features for members (blue) and non-members (orange). We compute five metrics across adjacent heads and layers: correlation, Frobenius distance, row-wise KL divergence, and barycenter drift (mean and variance). In all cases, the distributions for members and non-members are clearly separable, indicating that transition dynamics encode reliable membership signals.
• Figure 4: KDE plots of head-level correlation of attention distribution under perturbation for members (blue) and non-members (orange). Distributions are clearly separated, highlighting that correlation shifts provide strong signals for membership inference. Results are shown on the Arxiv subset of MIMIR with the Pythia-1.4B model.
• Figure 5: ROC AUC of AttenMIA when using perturbed attention features from different layers of Pythia-1.4B on the PubMed Central subset. We show results for individual layers (best layer value is highlighted). All layers contain exploitable membership signals, with performance improving when aggregating across groups and reaching its peak when using all layers together.