Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuning

Fangyuan Xu, Sihao Chen, Zinan Lin, Taiwei Shi, Sydney Graham, Pei Zhou, Mengting Wan, Alex Stein, Virginia Estellers, Charles Chen, Morris Sharp, Richard Speyer, Tadas Baltrusaitis, Jennifer Neville, Eunsol Choi, Longqi Yang

TL;DR

The experiments show that DP-RFT closes the gap between private evolution and DP finetuning methods in terms of the fidelity and downstream utility of the generated synthetic data, while respecting the private data boundary.

Abstract

Differentially private (DP) synthetic data generation plays a pivotal role in developing large language models (LLMs) on private data, where data owners cannot provide eyes-on access to individual examples. Generating DP synthetic data typically involves a difficult trade-off. On one hand, DP finetuning methods train an LLM as a synthetic data generator with formal privacy guarantees, yet it still requires the raw content of private examples for model training. However, methods that avoid direct exposure to private data are bounded by an off-the-shelf, un-finetuned model, whose outputs often lack domain fidelity. Can we train an LLM to generate high-quality synthetic text without eyes-on access to individual private examples? In this work, we introduce Differentially Private Reinforcement Fine-Tuning (DP-RFT), an online reinforcement learning algorithm for synthetic data generation with LLMs. DP-RFT leverages DP-protected nearest-neighbor votes from an eyes-off private corpus as a reward signal for on-policy synthetic samples generated by an LLM. The LLM iteratively learns to generate synthetic data to maximize the expected DP votes through Proximal Policy Optimization (PPO). We evaluate DP-RFT for long-form and domain-specific synthetic data generation, such as news articles, meeting transcripts, and medical article abstracts. Our experiments show that DP-RFT closes the gap between private evolution and DP finetuning methods in terms of the fidelity and downstream utility of the generated synthetic data, while respecting the private data boundary.

DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuning

TL;DR

The experiments show that DP-RFT closes the gap between private evolution and DP finetuning methods in terms of the fidelity and downstream utility of the generated synthetic data, while respecting the private data boundary.

Abstract

Differentially private (DP) synthetic data generation plays a pivotal role in developing large language models (LLMs) on private data, where data owners cannot provide eyes-on access to individual examples. Generating DP synthetic data typically involves a difficult trade-off. On one hand, DP finetuning methods train an LLM as a synthetic data generator with formal privacy guarantees, yet it still requires the raw content of private examples for model training. However, methods that avoid direct exposure to private data are bounded by an off-the-shelf, un-finetuned model, whose outputs often lack domain fidelity. Can we train an LLM to generate high-quality synthetic text without eyes-on access to individual private examples? In this work, we introduce Differentially Private Reinforcement Fine-Tuning (DP-RFT), an online reinforcement learning algorithm for synthetic data generation with LLMs. DP-RFT leverages DP-protected nearest-neighbor votes from an eyes-off private corpus as a reward signal for on-policy synthetic samples generated by an LLM. The LLM iteratively learns to generate synthetic data to maximize the expected DP votes through Proximal Policy Optimization (PPO). We evaluate DP-RFT for long-form and domain-specific synthetic data generation, such as news articles, meeting transcripts, and medical article abstracts. Our experiments show that DP-RFT closes the gap between private evolution and DP finetuning methods in terms of the fidelity and downstream utility of the generated synthetic data, while respecting the private data boundary.
Paper Structure (49 sections, 1 equation, 4 figures, 16 tables)

This paper contains 49 sections, 1 equation, 4 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Background and Related work
  3. Differential Privacy with LLMs.
  4. Reinforcement fine-tuning (RFT) for LLMs.
  5. DP-RFT: Differentially-Private Reinforcement Finetuning
  6. Generating input prompts
  7. Rewards
  8. DP voting as reward.
  9. Ensuring differential privacy.
  10. Mitigating reward hacking.
  11. Experiments
  12. Evaluation settings
  13. Datasets.
  14. Intrinsic evaluation: similarity with real data.
  15. Extrinsic evaluation: downstream performance.
  16. ...and 34 more sections

Figures (4)

  • Figure 1: An illustration of DP-RFT and comparison with prior methods. DP-RFT fine-tunes a language model (LM) to generate texts similar to the private document with reinforcement learning, guided by a DP-protected nearest-neighbor votes as reward. Unlike DP-Finetuning which requires accessing the private data directly, DP-RFT and Aug-PE allow downstream model training outside of the private boundary. While Aug-PE is bounded by samples generated from a frozen LM, DP-RFT generates synthetic data with higher fidelity by training an LM to generate texts more similar to the private data.
  • Figure 2: Procedure for computing similarity reward ($R_{sim}$) given a noise multiplier $\sigma$ for a batch of $s$ synthetic samples.
  • Figure 3: Analysis of structural similarity on WildChat. Top: Histogram of word-level maximum Jaccard similarity of synthetic WildChat examples against all private examples, comparing DP-RFT vs. Aug-PE under different privacy budgets. Bottom: Distribution of per-turn word counts from synthetic WildChat chatlogs generated by DP-RFT vs. Aug-PE under different privacy budgets ($\epsilon=\{1,2,3,\infty\}$). Real (private) WildChat examples are shown in gray.
  • Figure 4: LLM as a judge results for similarity measurement. We report both the win rate and the tie rate, included in the bracket. We compare samples generated by DP-RFT against those generated by Aug-PE under the same privacy budget ($\epsilon$).