Graphical vs. Deep Generative Models: Measuring the Impact of Differentially Private Mechanisms and Budgets on Utility

TL;DR

This work empirically compares graphical and deep DP generative models for tabular data to illuminate how privacy budgets are allocated and how this affects utility across high-dimensional datasets. By profiling budget spending across rows/columns and evaluating on scalability, statistics, similarity, clustering, and classification tasks, the study reveals that graphical models struggle to scale with dimensionality while deep models remain more scalable but exhibit mixed, task-dependent performance. Key findings show MST often best captures simple statistics but can overfit with more data, whereas PATE-GAN and DP-WGAN better handle complex correlations in high-dimensional settings, albeit with variability across datasets and privacy levels. The results yield practical guidance for selecting DP synthetic data models based on dataset dimensionality and downstream tasks, and they point to directions like dimensionality reduction and pre-training to improve DP data utility in real-world deployments.

Abstract

Generative models trained with Differential Privacy (DP) can produce synthetic data while reducing privacy risks. However, navigating their privacy-utility tradeoffs makes finding the best models for specific settings/tasks challenging. This paper bridges this gap by profiling how DP generative models for tabular data distribute privacy budgets across rows and columns, which is one of the primary sources of utility degradation. We compare graphical and deep generative models, focusing on the key factors contributing to how privacy budgets are spent, i.e., underlying modeling techniques, DP mechanisms, and data dimensionality. Through our measurement study, we shed light on the characteristics that make different models suitable for various settings and tasks. For instance, we find that graphical models distribute privacy budgets horizontally and thus cannot handle relatively wide datasets for a fixed training time; also, the performance on the task they were optimized for monotonically increases with more data but could also overfit. Deep generative models spend their budgets per iteration, so their behavior is less predictable with varying dataset dimensions, but are more flexible as they could perform better if trained on more features. Moreover, low levels of privacy ($ε\geq100$) could help some models generalize, achieving better results than without applying DP. We believe our work will aid the deployment of DP synthetic data techniques by navigating through the best candidate models vis-a-vis the dataset features, desired privacy levels, and downstream tasks.

Figures (28)

• Figure 1: T1: Off-diagonal pairwise correlation for different $\epsilon$ levels, on Corr Gauss, varying $n$ and $d$.
• Figure 2: T2: Marginal and pairwise mutual information similarity for different $\epsilon$ levels, on Census, varying $n$.
• Figure 3: T2: Marginal and pairwise mutual information similarity zoomed-in for MST and PATE-GAN for different $\epsilon$ levels and $n$ (Census).
• Figure 4: T2: Example fitted networks for PrivBayes (with network degree 3) and MST with $\epsilon=1$, on Census. The nodes correspond to the columns in the dataset, while the edges denote dependencies between them. For PrivBayes, the edges represent conditional distributions, for MST, they represent 2-way marginal counts; both are noisily measured to capture a collection of low-dimensional distributions and are used to generate synthetic data.
• Figure 5: T2: Pairwise mutual information (connected/non-connected nodes extracted from the fitted networks for PrivBayes and MST) similarity for different $\epsilon$ levels, on Census, varying $n$.