Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Just Fine-tune Twice: Selective Differential Privacy for Large Language Models

Weiyan Shi, Ryan Shea, Si Chen, Chiyuan Zhang, Ruoxi Jia, Zhou Yu

TL;DR

The paper tackles privacy leakage in large language models by extending Selective Differential Privacy (SDP) to transformer-based systems through Just Fine-tune Twice (JFT). JFT employs a two-phase approach: first redact-sensitive tokens with a policy-driven detector and fine-tune on the redacted data, then privately fine-tune on the original private data to achieve SDP, with analysis showing privacy budgets add up across phases. It introduces secret detectors at multiple levels (entity-level and contextual-level), and demonstrates that manual screening and privacy-amplified light-noise optimization can further improve utility under SDP, while empirical attacks (canary insertion) validate privacy protections. The results show robust utility across NLP tasks (GLUE and generation benchmarks) under varying detectors and budgets, offering a practical pathway to deploy SDP-enabled LLMs in real-world applications.

Abstract

Protecting large language models from privacy leakage is becoming increasingly crucial with their wide adoption in real-world products. Yet applying differential privacy (DP), a canonical notion with provable privacy guarantees for machine learning models, to those models remains challenging due to the trade-off between model utility and privacy loss. Utilizing the fact that sensitive information in language data tends to be sparse, Shi et al. (2021) formalized a DP notion extension called Selective Differential Privacy (SDP) to protect only the sensitive tokens defined by a policy function. However, their algorithm only works for RNN-based models. In this paper, we develop a novel framework, Just Fine-tune Twice (JFT), that achieves SDP for state-of-the-art large transformer-based models. Our method is easy to implement: it first fine-tunes the model with redacted in-domain data, and then fine-tunes it again with the original in-domain data using a private training mechanism. Furthermore, we study the scenario of imperfect implementation of policy functions that misses sensitive tokens and develop systematic methods to handle it. Experiments show that our method achieves strong utility compared to previous baselines. We also analyze the SDP privacy guarantee empirically with the canary insertion attack.

Just Fine-tune Twice: Selective Differential Privacy for Large Language Models

TL;DR

The paper tackles privacy leakage in large language models by extending Selective Differential Privacy (SDP) to transformer-based systems through Just Fine-tune Twice (JFT). JFT employs a two-phase approach: first redact-sensitive tokens with a policy-driven detector and fine-tune on the redacted data, then privately fine-tune on the original private data to achieve SDP, with analysis showing privacy budgets add up across phases. It introduces secret detectors at multiple levels (entity-level and contextual-level), and demonstrates that manual screening and privacy-amplified light-noise optimization can further improve utility under SDP, while empirical attacks (canary insertion) validate privacy protections. The results show robust utility across NLP tasks (GLUE and generation benchmarks) under varying detectors and budgets, offering a practical pathway to deploy SDP-enabled LLMs in real-world applications.

Abstract

Protecting large language models from privacy leakage is becoming increasingly crucial with their wide adoption in real-world products. Yet applying differential privacy (DP), a canonical notion with provable privacy guarantees for machine learning models, to those models remains challenging due to the trade-off between model utility and privacy loss. Utilizing the fact that sensitive information in language data tends to be sparse, Shi et al. (2021) formalized a DP notion extension called Selective Differential Privacy (SDP) to protect only the sensitive tokens defined by a policy function. However, their algorithm only works for RNN-based models. In this paper, we develop a novel framework, Just Fine-tune Twice (JFT), that achieves SDP for state-of-the-art large transformer-based models. Our method is easy to implement: it first fine-tunes the model with redacted in-domain data, and then fine-tunes it again with the original in-domain data using a private training mechanism. Furthermore, we study the scenario of imperfect implementation of policy functions that misses sensitive tokens and develop systematic methods to handle it. Experiments show that our method achieves strong utility compared to previous baselines. We also analyze the SDP privacy guarantee empirically with the canary insertion attack.
Paper Structure (23 sections, 2 theorems, 3 equations, 3 figures, 7 tables)

This paper contains 23 sections, 2 theorems, 3 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Jft: Just Fine-tune Twice
  4. Phase 1: Redacted-fine-tune
  5. Phase 2: Private-fine-tune.
  6. Privacy Analysis
  7. Secret Detectors of Different Levels
  8. Experiments
  9. Results
  10. Secret Detectors of Different Levels
  11. Selective Manual Screening
  12. Lightly Noised Optimizer with Privacy Amplification
  13. Attack results.
  14. Related Work
  15. Conclusions
  16. ...and 8 more sections

Key Result

Theorem 1

Given that 1) in the first phase, the data used for fine-tuning do not contain sensitive tokens and a public optimizer is used, and 2) in the second phase, the private optimizer achieves $(\epsilon,\delta)$-DP, Jft achieves $(\epsilon, \delta)$-SDP.

Figures (3)

  • Figure 1: The two-phase Jft mechanism. As pre-processing, we apply the secret detector to redact the private data $D$ and obtain the redacted data $D'$. Next, depending on the detector's performance, we use different ways to fine-tune the language model on the redacted $D'$ and obtain a redacted model. Then we fine-tune the model again on the private data $D$ with a private optimizer (e.g., DPSGD) to achieve an SDP-protected model.
  • Figure 2: Canary exposure for different models.
  • Figure 3: Exposure for different models when the canary is inserted only once.

Theorems & Definitions (8)

  • Definition 1: Differential Privacy
  • Definition 2: Policy Function
  • Definition 3
  • Definition 4
  • Theorem 1
  • Theorem 2
  • proof
  • proof