Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe

TL;DR

The paper presents a practical recipe for generating privacy-preserving synthetic text by fine-tuning pretrained language models with differential privacy (DP-SGD) and sampling with control codes to condition outputs. It demonstrates that DP-trained generators can yield synthetic data with downstream-Task utility comparable to non-private or DP-only baselines, while offering formal privacy protection and benefiting from DP post-processing to support unlimited downstream tasks without additional privacy loss. Comprehensive analyses on Yelp and private Microsoft feedback data reveal a consistent length-truncation effect under DP, strong resistance to private-information leakage via canary tests, and good distributional similarity to real data when larger models are used. The work argues for the practicality of DP synthetic text in real-world applications, while candidly discussing limitations around tail distribution learning and class-imbalance effects that warrant future improvements.

Abstract

Privacy concerns have attracted increasing attention in data-driven products due to the tendency of machine learning models to memorize sensitive training data. Generating synthetic versions of such data with a formal privacy guarantee, such as differential privacy (DP), provides a promising path to mitigating these privacy concerns, but previous approaches in this direction have typically failed to produce synthetic data of high quality. In this work, we show that a simple and practical recipe in the text domain is effective: simply fine-tuning a pretrained generative language model with DP enables the model to generate useful synthetic text with strong privacy protection. Through extensive empirical analyses on both benchmark and private customer data, we demonstrate that our method produces synthetic text that is competitive in terms of utility with its non-private counterpart, meanwhile providing strong protection against potential privacy leakages.

Figures (5)

• Figure 1: Illustration of our problem and methodology. We propose to generate synthetic text with a formal privacy guarantee: we fine-tune a generative language model with DP and then leverage it for synthetic text generation using control codes. Privacy loss of the overall procedure can be controlled by the data generation stage as, by the robustness to post-processing property of DP, the downstream task stage does not incur any additional privacy loss.
• Figure 2: Topic distributions of the synthetic and the original dataset are similar. The similarity further improves as the model size increases.
• Figure 3: Synthetic data generated w/ or w/o DP includes shorter sequences compared with the original data. This is more pronounced when the synthetic data is produced with DP, especially for the small model.
• Figure 4: Synthetic data generated with DP tends to be shorter compared to the data generated without DP. The plot shows sequence length distributions of the synthetic data generated with and without DP and the original customer feedback data.
• Figure 5: Distributions of perplexities of private information of injected canary sequences among their similar set of candidates measured by GPT2 models trained with and without DP. The dashed lines represent the perplexity of private information. Even a single-time occurring private information can achieve top rank in a non-private model which is not the case in the models trained with DP.

Theorems & Definitions (1)

• Definition 2.1: Differential Privacy (DP) dwork2006calibrating