How to DP-fy Your Data: A Practical Guide to Generating Synthetic Data With Differential Privacy

TL;DR

This paper surveys the landscape of differential privacy (DP) synthetic data across modalities (tabular, image, text) and federated settings, offering a practical framework for building end-to-end DP data-generation systems. It contrasts DP-training methods with training-free approaches, and workload-adaptive tabular techniques with workload-agnostic generative models, highlighting the trade-offs between utility, privacy, and computation. A central contribution is the organized taxonomy of DP synthetic data techniques, accompanied by principled guidance on privacy-unit selection, empirical privacy auditing, and lineage tracking to support real-world deployment. The work also synthesizes evaluation metrics—fidelity, utility, privacy leakage, and fairness—and outlines open questions and future directions, aiming to spur robust adoption, standardization, and further research in DP synthetic data. Overall, the paper provides a comprehensive, practitioner-friendly roadmap for designing, evaluating, and deploying DP synthetic data systems while emphasizing the importance of accountability, auditing, and data lineage in privacy-preserving AI.

Abstract

High quality data is needed to unlock the full potential of AI for end users. However finding new sources of such data is getting harder: most publicly-available human generated data will soon have been used. Additionally, publicly available data often is not representative of users of a particular system -- for example, a research speech dataset of contractors interacting with an AI assistant will likely be more homogeneous, well articulated and self-censored than real world commands that end users will issue. Therefore unlocking high-quality data grounded in real user interactions is of vital interest. However, the direct use of user data comes with significant privacy risks. Differential Privacy (DP) is a well established framework for reasoning about and limiting information leakage, and is a gold standard for protecting user privacy. The focus of this work, \emph{Differentially Private Synthetic data}, refers to synthetic data that preserves the overall trends of source data,, while providing strong privacy guarantees to individuals that contributed to the source dataset. DP synthetic data can unlock the value of datasets that have previously been inaccessible due to privacy concerns and can replace the use of sensitive datasets that previously have only had rudimentary protections like ad-hoc rule-based anonymization. In this paper we explore the full suite of techniques surrounding DP synthetic data, the types of privacy protections they offer and the state-of-the-art for various modalities (image, tabular, text and decentralized). We outline all the components needed in a system that generates DP synthetic data, from sensitive data handling and preparation, to tracking the use and empirical privacy testing. We hope that work will result in increased adoption of DP synthetic data, spur additional research and increase trust in DP synthetic data approaches.

Figures (5)

• Figure 1: Visual representation of choice of privacy unit for a toy private dataset of emails, reproduced with permission from ganesh2025itsdatatooprivate. Figure (a) treats the dataset as a flat collection of emails (example-level privacy). In this setting, secrets occurring in multiple emails are not protected. Figure (b) demonstrate the meaning of user-level privacy unit where each email is considered to belong only to the sender (single-attribution user-level privacy unit). User's secrets that are also present in emails the user received but didn't author might be not protected. Figure (c) attributes emails to all senders and receivers, and (d) bounds the contribution of each user to at most $k=2$ examples from (c) by selecting a subset of emails that will be used for training.
• Figure 2: DP synthetic data can be used as a drop-in replacement for the original private data and can be safely used for training downstream models, data analysis, human labeling and other applications. DP synthetic data is created either with the help of DP-Training of ML models or Training free methods.
• Figure 3: The select-measure-estimate paradigm. (1) A mechanism or domain expert selects a set of queries, possibly based on the workload and/or data. (2) The selected queries are measured by a standard noise addition mechanism like the Gaussian mechanism. (3) The data distribution is estimated from the noisy measurements. (4) The process optionally repeats from step (1) until the privacy budget is spent. (5) Synthetic data generated from the data distribution via a sampling or randomized rounding procedure.
• Figure 4: A typical cross-device federated learning system for mobile applications. The figure is adapted from kairouz2021advanceszhang23flgboard with permission. Private user data are decentralized on client devices, and focused updates are ephemerally aggregated in line with the data minimization principle bonawitz2022federated, an approach that can be further enhanced with techniques like encryption and differential privacy to achieve a multi-faceted model of privacy.
• Figure 5: Federated learning with differential privacy deploying Algorithm \ref{['alg:dpfl']}. The architecture is based on current production systems with on-device model training xu23gboard, and the integration of Trusted Execution Environments for enhanced scalability, verifiability, and privacy in future systems daly2024federated.

Theorems & Definitions (5)

• Definition 1: $(\epsilon,\delta)$-Differential Privacy
• Definition 2: $\ell_2$-sensitivity
• Remark 1: Workload vs. Measurements
• Remark 2: Workload-Adaptive vs. Workload-Agnostic
• Remark 3