User Inference Attacks on Large Language Models

TL;DR

<3-5 sentence high-level summary> The paper investigates privacy risks when fine-tuning large language models on user data and introduces the user inference threat model, where an attacker with black-box access uses a few samples from a user to infer whether that user contributed to fine-tuning. It proposes a practical likelihood-ratio attack that leverages a reference model and does not require access to the exact training data, and demonstrates strong attack performance across multiple domains (emails, news, and social media) including worst-case canaries. The study also analyzes factors driving vulnerability—such as the fraction of data a user contributes and the distinctiveness of a user’s data—and evaluates several mitigations (data limits, deduplication, early stopping, gradient clipping, and example-level DP) that offer partial protection but do not fully eliminate risk. The results underscore the need for scalable user-level privacy mechanisms, particularly DP at the user level, to safely deploy LLMs fine-tuned on user data.

Abstract

Fine-tuning is a common and effective method for tailoring large language models (LLMs) to specialized tasks and applications. In this paper, we study the privacy implications of fine-tuning LLMs on user data. To this end, we consider a realistic threat model, called user inference, wherein an attacker infers whether or not a user's data was used for fine-tuning. We design attacks for performing user inference that require only black-box access to the fine-tuned LLM and a few samples from a user which need not be from the fine-tuning dataset. We find that LLMs are susceptible to user inference across a variety of fine-tuning datasets, at times with near perfect attack success rates. Further, we theoretically and empirically investigate the properties that make users vulnerable to user inference, finding that outlier users, users with identifiable shared features between examples, and users that contribute a large fraction of the fine-tuning data are most susceptible to attack. Based on these findings, we identify several methods for mitigating user inference including training with example-level differential privacy, removing within-user duplicate examples, and reducing a user's contribution to the training data. While these techniques provide partial mitigation of user inference, we highlight the need to develop methods to fully protect fine-tuned LLMs against this privacy risk.

Figures (14)

• Figure 1: The user inference threat model. An LLM is fine-tuned on user-stratified data. The adversary can query samples on the fine-tuned model to compute likelihoods. The adversary can access samples from a user's distribution (different than the user training samples) to compute a likelihood score to determine if the user participated in training.
• Figure 2: Our attack can achieve significant AUROC, e.g., on the Enron emails dataset. Left three: Histograms of the test statistics for held-in and held-out users for the three attack evaluation datasets. Rightmost: Their corresponding ROC curves.
• Figure 3: Attack success over fine-tuning: User inference AUROC and the held-in/held-out validation loss.
• Figure 4: Attack success vs. model scale: User inference attack performance in $125$M and $1.3$B models trained on CC News. Left: Although the $1.3$B model achieves lower validation loss, the difference in validation loss between held-in and held-out users is the same as that of the $125$M model. Center & Right: User inference attacks against the $125$M and $1.3$B models achieve the same performance.
• Figure 5: Attack performance vs. attacker knowledge: As we increase the number of examples given to the attacker, the attack performance increases across all three datasets. The shaded area denotes the std over $100$ random draws of attacker examples.

Theorems & Definitions (3)

• Proposition 1
• Proposition 2: Mixing Distributions Brings Them Closer
• proof