PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMs

TL;DR

PII-Scope introduces a unified benchmark for evaluating PII extraction attacks on LLMs across diverse threat settings, providing a taxonomy of five attack types and a standardized evaluation protocol. The study demonstrates that single-query leakage estimates substantially underestimate real-world risk, while multi-query and continual attack scenarios can increase PII leakage up to about fivefold for email and reveal greater vulnerability in finetuned models. Using pretrained and finetuned GPT-J-6B and Pythia-6.9B (and LLaMa-7B with scrubbing), the authors show pervasive privacy risks across two PIIs (email and phone) and reveal hyperparameter sensitivities across hard-prompt, soft-prompt, and in-context strategies. The work offers a rigorous empirical foundation for defense development and model auditing, highlighting the need for robust privacy-preserving techniques and principled data-leakage evaluation sets.

Abstract

In this work, we introduce PII-Scope, a comprehensive benchmark designed to evaluate state-of-the-art methodologies for PII extraction attacks targeting LLMs across diverse threat settings. Our study provides a deeper understanding of these attacks by uncovering several hyperparameters (e.g., demonstration selection) crucial to their effectiveness. Building on this understanding, we extend our study to more realistic attack scenarios, exploring PII attacks that employ advanced adversarial strategies, including repeated and diverse querying, and leveraging iterative learning for continual PII extraction. Through extensive experimentation, our results reveal a notable underestimation of PII leakage in existing single-query attacks. In fact, we show that with sophisticated adversarial capabilities and a limited query budget, PII extraction rates can increase by up to fivefold when targeting the pretrained model. Moreover, we evaluate PII leakage on finetuned models, showing that they are more vulnerable to leakage than pretrained models. Overall, our work establishes a rigorous empirical benchmark for PII extraction attacks in realistic threat scenarios and provides a strong foundation for developing effective mitigation strategies.

Figures (33)

• Figure 1: Taxonomy of PII extraction attacks on LLMs. Note that the attacks designed for the black-box setting are also applicable to the white-box setting.
• Figure 2: Illustration of input prompt construction with different PII attacks. The attacker employs various strategies, including prompting the model with true prefixes carlini2021; using template prompts huang2022large; leveraging additional context from PII pairs (ICL) huang2022large, true prefixes of other data subjects (PII Compass) nakka2024pii; or learning soft prompt on a small subset containing PII pairs of a few data subjects kim2024propile.
• Figure 3: Template attack prompts for the sample data subject, Karen Arnold. These four template prompts are part of most of the previous PII leakage assessment works huang2022largewang2023decodingtrustsun2024trustllm.
• Figure 4: SPT attack pipeline kim2024propile. On the left, we train the soft prompt using the PII pairs in the adversary dataset $\mathcal{D}_\text{adv}$ by prepending the soft prompt to the template prompt embeddings of data subjects in $\mathcal{D}_\text{adv}$, and minimizing the cross-entropy loss with the objective of predicting the PII of the input data subject. On the right, the learned PII-evoking soft prompt embeddings are used to extract PIIs from other data subjects, such as those in $\mathcal{D}_\text{eval}$.
• Figure 5: Existing benchmark. Among the 3,333 data subjects in the original Enron PII leakage dataset huang2022large, there are only 404 unique email domains, indicating that many data subjects share the same domains. Here, we show the frequency of the top-30 most common email domains from the 404 domains, along with the cumulative proportion these data subjects constitute in the original dataset. We observe that just the top-30 domains alone account for 45% of the data subjects in the original dataset.