Harnessing large-language models to generate private synthetic text

TL;DR

This work tackles the challenge of sharing private data by generating differential-privacy (DP) protected synthetic text using a privately fine-tuned large language model (LLM). It introduces a data-synthesis pipeline that uses a public pre-trained decoder-only LLM, finetuned with DP training under a Prefix-LM objective, to produce DP synthetic datasets suitable for training downstream classifiers and for hyperparameter tuning. The study shows that parameter-efficient fine-tuning, especially LoRA, yields substantial gains over full fine-tuning in privacy-constrained settings, achieving downstream performance on par with or surpassing models trained directly with DP on real data. It also demonstrates that DP synthetic data can be freely shared and used for various purposes, including hyperparameter optimization, while maintaining strong privacy guarantees, and that standard proxy metrics like perplexity and MAUVE provide useful, though imperfect, estimates of synthetic-data quality. Overall, the approach offers a practical, scalable path to safe data sharing and model development in privacy-sensitive domains, with notable improvements over prior work and clear guidance for tuning under DP constraints.

Abstract

Differentially private training algorithms like DP-SGD protect sensitive training data by ensuring that trained models do not reveal private information. An alternative approach, which this paper studies, is to use a sensitive dataset to generate synthetic data that is differentially private with respect to the original data, and then non-privately training a model on the synthetic data. Doing so has several advantages: synthetic data can be reused for other tasks (including for hyper parameter tuning), retained indefinitely, and shared with third parties without sacrificing privacy. However, generating private synthetic data is much harder than training a private model. To improve performance on text data, recent work has utilized public data by starting with a pre-trained generative language model and privately fine-tuning it on sensitive data. This model can be used to sample a DP synthetic dataset. While this strategy seems straightforward, executing it has proven problematic. Previous approaches either show significant performance loss, or have, as we show, critical design flaws. In this paper we demonstrate that a proper training objective along with tuning fewer parameters results in excellent DP synthetic data quality. Our approach is competitive with direct DP-training of downstream classifiers in terms of performance on downstream tasks. Further, we demonstrate that our DP synthetic data is not only useful for downstream classifier training, but also to tune those same models.

Figures (7)

• Figure 1: Synthetic data generation and evaluation.
• Figure 2: Training loss, training accuracy and validation perplexity for the best prompt tuning training run with Adafactor optimizer, $\epsilon = 1$.
• Figure 3: Example of varying MAUVE parameters on estimating IMDB downstream performance on datasets differing in training epsilons from Table \ref{['tab:downstream_accuracy']}. Results are shown for the Sentence-T5-8B model.
• Figure 4: Proxy metrics for estimating dataset quality. Each point represents a metric's estimate of a synthesized dataset plotted against its true downstream classifier performance. X-axis shows the metric values re-scaled. All metrics are only useful to compare datasets, and thus their absoluteabsolute value is uninformative. Top Left: Different learning rates of an IMDB prompt-tuning model. Top Right: Estimating IMDB dataset quality for results in Table \ref{['tab:downstream_accuracy']}. Bottom Right: Estimating Yelp dataset quality for results in Table \ref{['tab:downstream_accuracy']}.
• Figure 5: Length distribution (in tokens) of IMDB synthetic data vs original dataset.

Theorems & Definitions (1)

• Definition 1: $(\epsilon, \delta)$-Differential Privacy, ODO