Scaling Up Membership Inference: When and How Attacks Succeed on Large Language Models

TL;DR

This work reframes membership inference attacks as a scalable problem for large language models by introducing multiscale benchmarks that span sentences to collections. It extends Dataset Inference methods with a two-stage signal aggregation to enable binary membership detection across scales, validated on The Pile and Pythia models. The results show that MIAs are notably effective at document and collection levels, with AUROC reaching up to around $0.9$ in favorable settings, especially under continual learning and end-task fine-tuning. The study provides practical benchmarks and highlights the privacy and copyright implications of MIA in real-world LLM deployments, suggesting that robust defenses will need to consider large-scale data aggregation and fine-tuning regimes.

Abstract

Membership inference attacks (MIA) attempt to verify the membership of a given data sample in the training set for a model. MIA has become relevant in recent years, following the rapid development of large language models (LLM). Many are concerned about the usage of copyrighted materials for training them and call for methods for detecting such usage. However, recent research has largely concluded that current MIA methods do not work on LLMs. Even when they seem to work, it is usually because of the ill-designed experimental setup where other shortcut features enable "cheating." In this work, we argue that MIA still works on LLMs, but only when multiple documents are presented for testing. We construct new benchmarks that measure the MIA performances at a continuous scale of data samples, from sentences (n-grams) to a collection of documents (multiple chunks of tokens). To validate the efficacy of current MIA approaches at greater scales, we adapt a recent work on Dataset Inference (DI) for the task of binary membership detection that aggregates paragraph-level MIA features to enable MIA at document and collection of documents level. This baseline achieves the first successful MIA on pre-trained and fine-tuned LLMs.

Figures (13)

• Figure 1: Focusing on the Right Scale. MIA has traditionally been considered ineffective for LLMs. However, we argue that MIA remains effective for LLMs when applied at a much larger scale, considering significantly longer token sequences. This large-scale MIA is also practically and legally relevant, as copyright is often determined at the document level.
• Figure 2: Preparing MIA Evaluation Datasets. Each source in The Pile is divided into Train, Dev, and Test splits, where the Train set is used for LLM training. For MIA, we designate the Train set as members and the Dev+Test sets as non-members. The MIA evaluation set ${t^1,\cdots,t^n}$ consists of datapoints for the binary detection task of predicting membership $m$. The benchmark makes some "known members" and "known non-members" available to the attackers; MIA methods may choose to use them or not. To support Collection-level MIA, we group documents in $n$ different ways to create member and non-member datasets.
• Figure 3: Effect of aggregation. We show MIA performances on arXiv at different levels of aggregation. Aggregation becomes more effective as we increase the number of aggregated instances.
• Figure 4: Failute modes for aggregation. Aggregation may not work when the base performance is too low (left and right plots), or when the amount of information to aggregate is too short (centre plot).
• Figure 5: Impact of paragraph-MIA performance after aggregation to dataset- and document-level MIA.