Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Differentially Private Synthetic Data Generation Algorithms

Yuchao Tao, Ryan McKenna, Michael Hay, Ashwin Machanavajjhala, Gerome Miklau

TL;DR

This work addresses the lack of systematic empirical evaluation of DP synthetic data generation for tabular data. It benchmarks GAN-based, marginal-based, and workload-based mechanisms across seven UCI datasets and multiple privacy budgets, using metrics that quantify marginal distributions, correlations, and ML classifier accuracy. The study finds that marginal-based methods, especially the MST approach, tend to outperform GAN-based methods, which often fail to preserve even simple statistics, and highlights the importance of discretization (PrivTree vs equal-width). These results inform practitioners about the most reliable approaches for preserving data utility under differential privacy and motivate future improvements in tabular GANs and preprocessing techniques. The framework and findings advance the evaluation of DP synthetic data, enabling more informed method selection in practice.

Abstract

This work presents a systematic benchmark of differentially private synthetic data generation algorithms that can generate tabular data. Utility of the synthetic data is evaluated by measuring whether the synthetic data preserve the distribution of individual and pairs of attributes, pairwise correlation as well as on the accuracy of an ML classification model. In a comprehensive empirical evaluation we identify the top performing algorithms and those that consistently fail to beat baseline approaches.

Benchmarking Differentially Private Synthetic Data Generation Algorithms

TL;DR

This work addresses the lack of systematic empirical evaluation of DP synthetic data generation for tabular data. It benchmarks GAN-based, marginal-based, and workload-based mechanisms across seven UCI datasets and multiple privacy budgets, using metrics that quantify marginal distributions, correlations, and ML classifier accuracy. The study finds that marginal-based methods, especially the MST approach, tend to outperform GAN-based methods, which often fail to preserve even simple statistics, and highlights the importance of discretization (PrivTree vs equal-width). These results inform practitioners about the most reliable approaches for preserving data utility under differential privacy and motivate future improvements in tabular GANs and preprocessing techniques. The framework and findings advance the evaluation of DP synthetic data, enabling more informed method selection in practice.

Abstract

This work presents a systematic benchmark of differentially private synthetic data generation algorithms that can generate tabular data. Utility of the synthetic data is evaluated by measuring whether the synthetic data preserve the distribution of individual and pairs of attributes, pairwise correlation as well as on the accuracy of an ML classification model. In a comprehensive empirical evaluation we identify the top performing algorithms and those that consistently fail to beat baseline approaches.
Paper Structure (7 sections, 4 figures, 2 tables)

This paper contains 7 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Mechanisms
  4. Datasets
  5. Metrics
  6. Findings
  7. Conclusion

Figures (4)

  • Figure 1: Overview of mechanisms in terms of optimal rate and average ranking across datasets, epsilons, and metrics stratified by metric groups. GT means "Grand Total."
  • Figure 2: Performance metrics for synthetic data algorithms at $\epsilon=1.0$
  • Figure 3: One-way marginal distributions for the original Adult dataset and for a sample synthetic dataset generated by each algorithm ($\epsilon=1.0$) in descending order by similarity (1-TVD, shown to the right of each row).
  • Figure 4: Correlation heatmaps for all pairwise categorical attributes from the Adult dataset. A heatmap is shown for the original data, Ground Truth (center), and for a sample synthetic dataset generated by each algorithm at $\epsilon=1.0$. Attributes are sorted by domain sizes.