Can Differentially Private Fine-tuning LLMs Protect Against Privacy Attacks?

TL;DR

This paper investigates whether differentially private fine‑tuning can mitigate privacy attacks on LLMs across both full and parameter‑efficient fine‑tuning (PEFT) methods. It systematically evaluates DP via a Book Keeping DP mechanism against two empirical attacks—data extraction and SPV‑MIA—across GPT‑2 and GPT‑2 XL, varying privacy budgets. The study finds that DP generally reduces empirical privacy risk but degrades model utility, with the severity of utility loss depending on the fine‑tuning method; full fine‑tuning offers the best privacy‑utility balance under DP, while LoRA and P‑tuning trade off utility and privacy differently, and prefix‑tuning performs poorly under DP. The results provide practical guidance for privacy‑sensitive deployments and highlight directions for optimizing the DP‑utility trade‑off in future PEFT research.

Abstract

Fine-tuning large language models (LLMs) has become an essential strategy for adapting them to specialized tasks; however, this process introduces significant privacy challenges, as sensitive training data may be inadvertently memorized and exposed. Although differential privacy (DP) offers strong theoretical guarantees against such leakage, its empirical privacy effectiveness on LLMs remains unclear, especially under different fine-tuning methods. In this paper, we systematically investigate the impact of DP across fine-tuning methods and privacy budgets, using both data extraction and membership inference attacks to assess empirical privacy risks. Our main findings are as follows: (1) Differential privacy reduces model utility, but its impact varies significantly across different fine-tuning methods. (2) Without DP, the privacy risks of models fine-tuned with different approaches differ considerably. (3) When DP is applied, even a relatively high privacy budget can substantially lower privacy risk. (4) The privacy-utility trade-off under DP training differs greatly among fine-tuning methods, with some methods being unsuitable for DP due to severe utility degradation. Our results provide practical guidance for privacy-conscious deployment of LLMs and pave the way for future research on optimizing the privacy-utility trade-off in fine-tuning methodologies.

Figures (5)

• Figure 1: Perplexity as a function of the privacy budget $\epsilon$ for various fine-tuning methods. The x-axis represents the privacy budget, while the y-axis shows the model perplexity in ln scale (with lower values indicating higher utility). Trends in the figure illustrate the impact of different DP budgets on model utility across various fine-tuning methods.
• Figure 2: Exposure as a function of the privacy budget $\epsilon$ for different fine-tuning methods. The four subplots display results for two model sizes and two attack scenarios. The top two subplots correspond to GPT-2, while the bottom two correspond to GPT-2 XL. The left subplots represent the weak attack (Short Prefix), and the right subplots represent the strong attack (Long Prefix). These trends illustrate how DP training affects model exposure across varying privacy budgets and attack strengths.
• Figure 3: MIA attack performance as a function of privacy budget $\epsilon$ on GPT-2 XL. This plot illustrates how the effectiveness of membership inference attacks (as measured by AUC) varies with different privacy budgets. Each point represents the MIA performance under a specific $\epsilon$ value.
• Figure 4: ROC Curves for Membership Inference Attacks (MIA) on Models with and without Differential Privacy (DP) Training. The top two subplots show the ROC curves for full fine-tuning (FFT) before DP training and after DP training (with $\epsilon$=10), while the bottom two subplots depict the corresponding results for LoRA. The ASR (Attack Success Rate) is indicated in the plots. This figure demonstrates that DP training significantly mitigates the effectiveness of MIA attacks, as reflected in the substantial degradation of the ROC curves when DP is applied.
• Figure 5: Trade-off between model privacy risk and utility. The left panel plots exposure versus perplexity, while the right panel shows AUC versus perplexity, with perplexity on the vertical axis in both plots. In each curve, the rightmost point corresponds to the model trained without differential privacy. We have enlarged part of the right panel to better observe the relationship between the curves.

Theorems & Definitions (4)

• remark thmcounterremark
• remark thmcounterremark
• remark thmcounterremark
• remark thmcounterremark