ACTG-ARL: Differentially Private Conditional Text Generation with RL-Boosted Control

TL;DR

The paper tackles the challenge of generating high-quality differentially private (DP) synthetic text with fine-grained control. It introduces ACTG-ARL, a hierarchical framework that decouples feature learning from conditional text generation and adds Anchored RL to strengthen instruction-following without sacrificing data fidelity. The key contributions are the ACTG framework (with a optimal configuration using a rich structured tabular feature, AIM for feature synthesis, and DP-FT for conditional generation) and the Anchored RL post-training recipe (hybrid RL+SFT objective with best-of-N anchors) that achieves strong per-instance control while preserving text quality. Empirical results on bioRxiv and PMC-patients show state-of-the-art DP text quality (MAUVE) and attribute distribution matching, plus improved instruction-following, enabling controllable, private generation for downstream analyses.

Abstract

Generating high-quality synthetic text under differential privacy (DP) is critical for training and evaluating language models without compromising user privacy. Prior work on synthesizing DP datasets often fail to preserve key statistical attributes, suffer utility loss from the noise required by DP, and lack fine-grained control over generation. To address these challenges, we make two contributions. First, we introduce a hierarchical framework that decomposes DP synthetic text generation into two subtasks: feature learning and conditional text generation. This design explicitly incorporates learned features into the generation process and simplifies the end-to-end synthesis task. Through systematic ablations, we identify the most effective configuration: a rich tabular schema as feature, a DP tabular synthesizer, and a DP fine-tuned conditional generator, which we term ACTG (Attribute-Conditioned Text Generation). Second, we propose Anchored RL (ARL), a post-training method that improves the instruction-following ability of ACTG for conditional generation. ARL combines RL to boost control with an SFT anchor on best-of-$N$ data to prevent reward hacking. Together, these components form our end-to-end algorithm ACTG-ARL, which advances both the quality of DP synthetic text (+20% MAUVE over prior work) and the control of the conditional generator under strong privacy guarantees.

Figures (23)

• Figure 1: Our hierarchical framework for DP synthetic text generation.
• Figure 2: End-to-end and modular evaluation of our hierarchical framework.(Rows 1–2) End-to-end comparison of our approaches with baselines (Aug-PE, vanilla DP-FT, CTCL) on bioRxiv and PMC-Patients, evaluated on fidelity (MAUVE, $d_{\text{JS}}^f$) and utility (classification F1, NTP accuracy). We omit Aug-PE on PMC-Patients in Row 2 (see full results in \ref{['app:results-pe-pmc']}) as its performance is substantially lower than other methods. (Row 3) Modular ablations and fine-grained error analysis for $\mathcal{S}_3$. Arrows in the figure titles indicate whether higher ($\uparrow$) or lower ($\downarrow$) values are better.
• Figure 3: Fine-grained error analysis in our framework.
• Figure 4: (a) IFAcc of the conditional generator $G_{x \mid f}$ with and without DP, showing a substantial drop under DP. (b) MAUVE score of generated text after RL, demonstrating a sharp decline in textual fidelity. (c) Example generation from the bioRxiv dataset that perfectly satisfies the input requirement (score: 8/8; see \ref{['app:results-reward-hacking-example']}) but fails to match the target domain (paper abstract). This occurs during RL training, where the model exploits the rubric reward and exhibits reward hacking.
• Figure 5: Performance of the conditional generators before and after RL evaluated on three metrics. ACTG-RL improves IFAcc but suffers from reward hacking, which collapses textual fidelity (MAUVE). ACTG-ARL resolves this trade-off, boosting IFAcc close to the non-DP level while maintaining high MAUVE and achieving the best attribute distribution matching.

Theorems & Definitions (4)

• Definition 2.1: $(\varepsilon, \delta)$-DP
• Remark 3.1
• Remark 3.2
• Remark 3.3