A Statistical and Multi-Perspective Revisiting of the Membership Inference Attack in Large Language Models

TL;DR

This work addresses the inconsistent performance of membership inference attacks on large language models by conducting thousands of experiments across multiple settings, domains, and model sizes. It systematically analyzes baseline, token-distribution, text-alteration, and black-box MIAs, revealing that most methods offer limited gain over baselines, but notable outliers exist and depend on domain and model size. The study also highlights thresholds as a critical, often overlooked factor, and connects MIA signals to text length, text similarity, embedding separability, and decoding entropy dynamics. These findings suggest that MIA effectiveness is nuanced and context-dependent, with implications for data privacy and monitoring in real-world LLM deployments.

Abstract

The lack of data transparency in Large Language Models (LLMs) has highlighted the importance of Membership Inference Attack (MIA), which differentiates trained (member) and untrained (non-member) data. Though it shows success in previous studies, recent research reported a near-random performance in different settings, highlighting a significant performance inconsistency. We assume that a single setting doesn't represent the distribution of the vast corpora, causing members and non-members with different distributions to be sampled and causing inconsistency. In this study, instead of a single setting, we statistically revisit MIA methods from various settings with thousands of experiments for each MIA method, along with study in text feature, embedding, threshold decision, and decoding dynamics of members and non-members. We found that (1) MIA performance improves with model size and varies with domains, while most methods do not statistically outperform baselines, (2) Though MIA performance is generally low, a notable amount of differentiable member and non-member outliers exists and vary across MIA methods, (3) Deciding a threshold to separate members and non-members is an overlooked challenge, (4) Text dissimilarity and long text benefit MIA performance, (5) Differentiable or not is reflected in the LLM embedding, (6) Member and non-members show different decoding dynamics.

Figures (10)

• Figure 1: Sample with different settings may result in MIA performance inconsistency.
• Figure 2: ROC-AUC probability density in different dimensions while fixing other dimensions. Less area on the left side means statistically better MIA performance. Shade area means variance from random seeds. We only enlarge Figure (d) to increase the readability due to the number of MIA methods.
• Figure 3: Boxplot of the threshold for different MIA methods over domains and model sizes.
• Figure 4: MIA outliers overlap matrix across methods.
• Figure 5: The DB Score (solid line with triangles) and Transformer Classifier Accuracy (dotted line with circles) on the member and non-member embeddings. The differentiable outliers come from DM Math, GitHub, and WikiMIA. The indifferentiable splits come from arXiv, Pile-CC, and PubMed.