Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Selective Differential Privacy for Language Modeling

Weiyan Shi, Aiqi Cui, Evan Li, Ruoxi Jia, Zhou Yu

TL;DR

The paper introduces selective differential privacy (S-DP) to protect only the sensitive portions of language data, addressing utility loss in traditional differential privacy. It defines a policy function to identify sensitive tokens and presents Selective-DPSGD, a training mechanism that applies DPSGD to private parts while updating non-private parts with SGD, including noise injected into hidden states. Empirical results on WikiText-2 and a synthetic dialog dataset (CustomerSim) show improved utility at comparable privacy budgets compared to canonical DP, with demonstrated robustness to standard privacy attacks. The work also analyzes imperfect policy functions and extends preliminary results to contextual policies and Transformer models, highlighting practical implications for privacy-preserving NLP.

Abstract

With the increasing applications of language models, it has become crucial to protect these models from leaking private information. Previous work has attempted to tackle this challenge by training RNN-based language models with differential privacy guarantees. However, applying classical differential privacy to language models leads to poor model performance as the underlying privacy notion is over-pessimistic and provides undifferentiated protection for all tokens in the data. Given that the private information in natural language is sparse (for example, the bulk of an email might not carry personally identifiable information), we propose a new privacy notion, selective differential privacy, to provide rigorous privacy guarantees on the sensitive portion of the data to improve model utility. To realize such a new notion, we develop a corresponding privacy mechanism, Selective-DPSGD, for RNN-based language models. Besides language modeling, we also apply the method to a more concrete application--dialog systems. Experiments on both language modeling and dialog system building show that the proposed privacy-preserving mechanism achieves better utilities while remaining safe under various privacy attacks compared to the baselines. The data and code are released at https://github.com/wyshi/lm_privacy to facilitate future research .

Selective Differential Privacy for Language Modeling

TL;DR

The paper introduces selective differential privacy (S-DP) to protect only the sensitive portions of language data, addressing utility loss in traditional differential privacy. It defines a policy function to identify sensitive tokens and presents Selective-DPSGD, a training mechanism that applies DPSGD to private parts while updating non-private parts with SGD, including noise injected into hidden states. Empirical results on WikiText-2 and a synthetic dialog dataset (CustomerSim) show improved utility at comparable privacy budgets compared to canonical DP, with demonstrated robustness to standard privacy attacks. The work also analyzes imperfect policy functions and extends preliminary results to contextual policies and Transformer models, highlighting practical implications for privacy-preserving NLP.

Abstract

With the increasing applications of language models, it has become crucial to protect these models from leaking private information. Previous work has attempted to tackle this challenge by training RNN-based language models with differential privacy guarantees. However, applying classical differential privacy to language models leads to poor model performance as the underlying privacy notion is over-pessimistic and provides undifferentiated protection for all tokens in the data. Given that the private information in natural language is sparse (for example, the bulk of an email might not carry personally identifiable information), we propose a new privacy notion, selective differential privacy, to provide rigorous privacy guarantees on the sensitive portion of the data to improve model utility. To realize such a new notion, we develop a corresponding privacy mechanism, Selective-DPSGD, for RNN-based language models. Besides language modeling, we also apply the method to a more concrete application--dialog systems. Experiments on both language modeling and dialog system building show that the proposed privacy-preserving mechanism achieves better utilities while remaining safe under various privacy attacks compared to the baselines. The data and code are released at https://github.com/wyshi/lm_privacy to facilitate future research .

Paper Structure

This paper contains 21 sections, 2 theorems, 7 equations, 8 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Backgrounds
  4. Selective Differential Privacy
  5. Selective Privacy Mechanism
  6. Experiments
  7. Evaluation
  8. Attack Details
  9. WikiText-2 Results
  10. CustomerSim Results
  11. Data Anonymization and Selective-DP
  12. S-DP achieves better utility
  13. Imperfect Policy Function
  14. Preliminary Results on Contextual Policy Function and Transformer
  15. Conclusions
  16. ...and 6 more sections

Key Result

Theorem 1

Assume that $\max_{x,w}\|g(x,w)\|\leq C$. Then, for any arbitrary $w$, adding Gaussian noise $\Delta =\mathcal{N}(0,\sigma^2)$ proportional to $C$ into $g$ can ensure $(\epsilon,\delta)$-DP where $\epsilon,\delta$ depends on $C$ and $\sigma$. More formally, for all neighboring datasets $x$ and $x'$

Figures (8)

  • Figure 1: "Data anonymization" and training with "No DP" cannot provide knobs to adjust the privacy-utility trade-off. Selective-DP improves the privacy-utility trade-off of traditional DP, to get closer to the ideal model with both high privacy and high utility.
  • Figure 2: All private variables are in red. We apply regular SGD on non-private variables and DPSGD on private variables in Selective-DPSGD.
  • Figure 3: Privacy-utility trade-off, canary insertion attack and membership inference attack on WikiText-2.
  • Figure 4: Privacy-utility trade-off, canary insertion attack and membership inference attack on CustomerSim.
  • Figure 5: Perplexity on private tokens over $\epsilon$ for data anonymization and S-DPSGD.
  • ...and 3 more figures

Theorems & Definitions (8)

  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4
  • Definition 5
  • Theorem 1
  • Theorem 2
  • proof