Does fine-tuning GPT-3 with the OpenAI API leak personally-identifiable information?

TL;DR

This work investigates whether fine-tuning GPT-3 via OpenAI's API leaks personally identifiable information (PII) from the training data. Using Enron emails, it conducts two attacks—Classification and Autocomplete—under black-box access to measure memorization of PII. Classification recovered 256 unique PIIs with a recall of 4.06% and precision of 2.45%, while Autocomplete leaked hundreds of PIIs with recalls around 26–28% and precision up to 13.16%, including OOD prompts. The findings reveal that API-based fine-tuning can expose sensitive data, underscoring the need for privacy-preserving fine-tuning techniques and rigorous privacy testing for enterprise deployments.

Abstract

Machine learning practitioners often fine-tune generative pre-trained models like GPT-3 to improve model performance at specific tasks. Previous works, however, suggest that fine-tuned machine learning models memorize and emit sensitive information from the original fine-tuning dataset. Companies such as OpenAI offer fine-tuning services for their models, but no prior work has conducted a memorization attack on any closed-source models. In this work, we simulate a privacy attack on GPT-3 using OpenAI's fine-tuning API. Our objective is to determine if personally identifiable information (PII) can be extracted from this model. We (1) explore the use of naive prompting methods on a GPT-3 fine-tuned classification model, and (2) we design a practical word generation task called Autocomplete to investigate the extent of PII memorization in fine-tuned GPT-3 within a real-world context. Our findings reveal that fine-tuning GPT3 for both tasks led to the model memorizing and disclosing critical personally identifiable information (PII) obtained from the underlying fine-tuning dataset. To encourage further research, we have made our codes and datasets publicly available on GitHub at: https://github.com/albertsun1/gpt3-pii-attacks

Figures (2)

• Figure 1: We run two experiments, Classification and Autocomplete, to detect memorization of sensitive personally identifiable information (PII) by fine-tuned GPT-3.
• Figure 2: Step-by-step flowchart of our simulated attacks for a GPT-3 (curie) model trained on the Enron Email dataset using the OpenAI fine-tuning API.