Do Membership Inference Attacks Work on Large Language Models?

TL;DR

This work systematically evaluates membership inference attacks on pretraining data for large language models, introducing the Mimir benchmark to unify evaluation across multiple MIAs. Across diverse domains and model sizes up to 12B parameters, MIAs largely perform near random, with notable leakage only in certain conditions tied to data distribution shifts and domain overlap. The authors identify two contributing factors: the scale of pretraining data with near-one-epoch training reducing memorization signals, and intrinsic ambiguity from high lexical overlap between members and non-members, which challenges traditional MIA definitions. They argue for rethinking membership in the context of generative models and propose semantic, fuzzy membership notions, along with releasing open-source tooling to advance privacy research in LLMs.

Abstract

Membership inference attacks (MIAs) attempt to predict whether a particular datapoint is a member of a target model's training data. Despite extensive research on traditional machine learning models, there has been limited work studying MIA on the pre-training data of large language models (LLMs). We perform a large-scale evaluation of MIAs over a suite of language models (LMs) trained on the Pile, ranging from 160M to 12B parameters. We find that MIAs barely outperform random guessing for most settings across varying LLM sizes and domains. Our further analyses reveal that this poor performance can be attributed to (1) the combination of a large dataset and few training iterations, and (2) an inherently fuzzy boundary between members and non-members. We identify specific settings where LLMs have been shown to be vulnerable to membership inference and show that the apparent success in such settings can be attributed to a distribution shift, such as when members and non-members are drawn from the seemingly identical domain but with different temporal ranges. We release our code and data as a unified benchmark package that includes all existing MIAs, supporting future work.

Figures (16)

• Figure 1: MIA performance as model size increases for the reference-based attack over select domains. We also plot the AUC ROC trajectory against the non-deduped Pythia suite for comparison. Increasing model size slightly boosts MIA performance while deduplication decreases performance. Other attacks follow similar trends (Appendix \ref{['fig:gen_exp_model_size+dedup_others']}).
• Figure 2: (Left) Reference-based attack performance as the amount of training data seen, measured in the number of training steps, increases across 1 epoch of the deduplicated Pile. In general, performance spikes greatly before gradually decreasing as the amount of training data seen increases. Other attacks (\ref{['fig:training-data-size-others']}, Appendix) follow similar trends. (Right) MIA performance on target model Datablations as the number of effective epochs increases via increasing epoch count. Performance increases linearly with the number of effective epochs. See \ref{['fig:num-epochs-silo']} for results on SILO.
• Figure 3: Natural distributions of 7-gram overlap of non-member data over select domains. Github has a considerably higher overlap than other domains.
• Figure 3: AUC-ROC on the temporally shifted Wikipedia benchmark across various MIAs. Target models are the Pythia-dedup suite models. For each model, the highest score across MIAs is bolded.
• Figure 4: Distribution of 7-gram overlap for the original and temporally-shifted non-members.