Differentially Private Language Models Benefit from Public Pre-training

TL;DR

The paper tackles the challenge of privacy-preserving language modeling by pretraining a base model on public data and then privately fine-tuning it on a private corpus using DP-SGD. It demonstrates that DP fine-tuning can significantly improve private-domain perplexity and make DP training feasible, even when training data is limited or out-of-distribution. The results across small and large feedforward architectures show that pre-training confers substantial benefits for privacy-enabled models, though performance remains below state-of-the-art non-private baselines. The work suggests a practical path for deploying high-quality, privacy-protecting language models and points to future work with more advanced architectures and longer training regimes.

Abstract

Language modeling is a keystone task in natural language processing. When training a language model on sensitive information, differential privacy (DP) allows us to quantify the degree to which our private data is protected. However, training algorithms which enforce differential privacy often lead to degradation in model quality. We study the feasibility of learning a language model which is simultaneously high-quality and privacy preserving by tuning a public base model on a private corpus. We find that DP fine-tuning boosts the performance of language models in the private domain, making the training of such models possible.

Figures (4)

• Figure 1: Test-set perplexity as a function of training iterations for the small (a) and large (b) language models. The legend indicates train-set / evaluation set, with $\sigma$ being the noise scale used in differentially private training. The fine-tuned models are trained on the Brown corpus and tuned on the Reddit dataset. The graph for $\sigma = 1.1$ for the large language model is not visible since all perplexity values are infinity. Note: the graphs are truncated to the first epoch of training. Perplexities change marginally after this point.
• Figure 2: $(\epsilon, \delta)$-privacy guarantees for $q = 10^{-3}, T = 10^5$, computed using the moments accountant 2016arXiv160700133A. Here, $\sigma$ is a noise-scale parameter specified by the user. This helps us to select a noise scale appropriate to a given application setting.
• Figure 3: The number of tokens in the training and test set of each dataset. Since we don't test on Brown, this entry is left empty.
• Figure 4: We provide the trade off between $\epsilon$ and test perplexity for the small and large models from figure \ref{['fig:exps_small_large']}. We hold $\delta$ to $1e-5$ and set the gradient clipping to $1.0$. We include the lowest test perplexity for each model. Recall the large model with $\sigma=1.1$ never converged to finite perplexity and is denoted NA.