Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Differentially Private Language Models for Secure Data Sharing

Justus Mattern, Zhijing Jin, Benjamin Weggenmann, Bernhard Schoelkopf, Mrinmaya Sachan

TL;DR

This work presents a globally differentially private data-release framework that trains a large language model on sensitive data with a DP optimizer and then samples a synthetic, attribute-controlled dataset to serve as a private 'twin' of the original data. By employing a prompt-based conditioning scheme and a prompt-mismatch objective, the approach enables controllable text generation with reduced mislabeled samples while preserving privacy. Empirical results on IMDb and Amazon reviews show that synthetic data achieves near real-data utility under DP budgets ($\epsilon$ = 3 or 8), and can outperform classifiers trained directly with DP-SGD on real data, even in few-shot settings. The study also evaluates data leakage and language quality, finding substantial privacy protection and high text quality, with practical implications for secure data sharing and downstream NLP tasks.

Abstract

To protect the privacy of individuals whose data is being shared, it is of high importance to develop methods allowing researchers and companies to release textual data while providing formal privacy guarantees to its originators. In the field of NLP, substantial efforts have been directed at building mechanisms following the framework of local differential privacy, thereby anonymizing individual text samples before releasing them. In practice, these approaches are often dissatisfying in terms of the quality of their output language due to the strong noise required for local differential privacy. In this paper, we approach the problem at hand using global differential privacy, particularly by training a generative language model in a differentially private manner and consequently sampling data from it. Using natural language prompts and a new prompt-mismatch loss, we are able to create highly accurate and fluent textual datasets taking on specific desired attributes such as sentiment or topic and resembling statistical properties of the training data. We perform thorough experiments indicating that our synthetic datasets do not leak information from our original data and are of high language quality and highly suitable for training models for further analysis on real-world data. Notably, we also demonstrate that training classifiers on private synthetic data outperforms directly training classifiers on real data with DP-SGD.

Differentially Private Language Models for Secure Data Sharing

TL;DR

This work presents a globally differentially private data-release framework that trains a large language model on sensitive data with a DP optimizer and then samples a synthetic, attribute-controlled dataset to serve as a private 'twin' of the original data. By employing a prompt-based conditioning scheme and a prompt-mismatch objective, the approach enables controllable text generation with reduced mislabeled samples while preserving privacy. Empirical results on IMDb and Amazon reviews show that synthetic data achieves near real-data utility under DP budgets ( = 3 or 8), and can outperform classifiers trained directly with DP-SGD on real data, even in few-shot settings. The study also evaluates data leakage and language quality, finding substantial privacy protection and high text quality, with practical implications for secure data sharing and downstream NLP tasks.

Abstract

To protect the privacy of individuals whose data is being shared, it is of high importance to develop methods allowing researchers and companies to release textual data while providing formal privacy guarantees to its originators. In the field of NLP, substantial efforts have been directed at building mechanisms following the framework of local differential privacy, thereby anonymizing individual text samples before releasing them. In practice, these approaches are often dissatisfying in terms of the quality of their output language due to the strong noise required for local differential privacy. In this paper, we approach the problem at hand using global differential privacy, particularly by training a generative language model in a differentially private manner and consequently sampling data from it. Using natural language prompts and a new prompt-mismatch loss, we are able to create highly accurate and fluent textual datasets taking on specific desired attributes such as sentiment or topic and resembling statistical properties of the training data. We perform thorough experiments indicating that our synthetic datasets do not leak information from our original data and are of high language quality and highly suitable for training models for further analysis on real-world data. Notably, we also demonstrate that training classifiers on private synthetic data outperforms directly training classifiers on real data with DP-SGD.
Paper Structure (28 sections, 5 equations, 2 figures, 7 tables)

This paper contains 28 sections, 5 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Differential Privacy
  4. Differentially Private Optimization
  5. Approach
  6. Conditional text generation with natural language prompts
  7. Reducing faulty labels with prompt-mismatch objective
  8. Evaluation
  9. Datasets
  10. IMDb Movie Reviews
  11. Amazon Multi Domain Reviews
  12. Implementation Details
  13. Experimental Results
  14. Data Utility
  15. Synthetic data is almost on par with real data:
  16. ...and 13 more sections

Figures (2)

  • Figure 1: Main idea of our paper: To share potentially sensitive datasets with third parties, we train a language model (LM) on the sensitive data in a differentially private manner and consequently prompt the LM to generate synthetic samples with privacy guarantees.
  • Figure 2: Our template-based approach for generating task instructions. A template consists of placeholders for verbalizations of different attribute values.

Theorems & Definitions (1)

  • Definition 1