Provably Protecting Fine-Tuned LLMs from Training Data Extraction
Tom Segal, Asaf Shabtai, Yuval Elovici
TL;DR
This work tackles privacy risks from training data extraction in fine-tuned LLMs by introducing SpaRPS, a property describing sparse relative probability shifts, and SCP-$\Delta_r$, a NAF-based defense operating on relative probabilities with base-model smoothing. By enforcing SpaRPS, SCP-$\Delta_r$ achieves substantially tighter protection bounds than traditional CP-based NAF methods and translates small theoretical guarantees into strong empirical resistance against TDE attacks, while preserving downstream utility. The authors provide both theoretical analyses and extensive empirical evaluations, including canary, PII, and token-by-token extraction attacks, demonstrating significant improvements over prior defenses. The approach offers practical, provable privacy guarantees for fine-tuning scenarios and establishes a foundation for extending similar protections to broader training regimes, with implications for safer deployment of private-data fine-tuning in real-world LLM applications.
Abstract
Fine-tuning large language models (LLMs) on sensitive datasets raises privacy concerns, as training data extraction (TDE) attacks can expose highly confidential information. Existing defenses against such attacks either lack formal privacy guarantees or incur substantial utility degradation. We observe that fine-tuning induces widespread probability shifts, yet preserving only a small subset of influential token-level deviations is sufficient; the remaining shifts can be aggressively smoothed with minimal impact on utility. Motivated by this insight, we propose SCP-$Δ_r$, a Near Access Freeness (NAF)-based algorithm that operates on relative probabilities and explicitly smooths low-impact tokens using a base model. SCP-$Δ_r$ achieves orders-of-magnitude better theoretical bounds than existing NAF based methods and provides strong empirical protection against TDE attacks with minimal performance loss.