The Real Deal Behind the Artificial Appeal: Inferential Utility of Tabular Synthetic Data

TL;DR

This work investigates the inferential utility of tabular synthetic data, showing that treating synthetic samples as real observations yields inflated type I error and overly optimistic confidence intervals. It analyzes bias, SE, and convergence across statistical and deep-learning generators, highlighting that DL-based synthetic data introduce extra variability and regularisation bias that slow convergence and understate SE. A minimal SE correction from Raab et al. partially mitigates the issue for parametric generators but fails to capture DL-induced uncertainty, as demonstrated by simulations and a case study on the Adult dataset. The findings imply that inference on synthetic data should be generator-aware, favor parametric generation when inference is the goal, and motivate development of DL-aware inferential tools to maintain valid uncertainty quantification.

Abstract

Recent advances in generative models facilitate the creation of synthetic data to be made available for research in privacy-sensitive contexts. However, the analysis of synthetic data raises a unique set of methodological challenges. In this work, we highlight the importance of inferential utility and provide empirical evidence against naive inference from synthetic data, whereby synthetic data are treated as if they were actually observed. Before publishing synthetic data, it is essential to develop statistical inference tools for such data. By means of a simulation study, we show that the rate of false-positive findings (type 1 error) will be unacceptably high, even when the estimates are unbiased. Despite the use of a previously proposed correction factor, this problem persists for deep generative models, in part due to slower convergence of estimators and resulting underestimation of the true standard error. We further demonstrate our findings through a case study.

Figures (11)

• Figure 1: General experimental framework, applied in both the simulation study and case study.
• Figure 2: DAG for the variables in the simulation study.
• Figure 3: The horizontal dashed line represents the population parameter and each dot is an estimate per Monte Carlo run (200 dots in total per value of $n$). The dashed funnel indicates the behaviour of an unbiased and $\sqrt{n}$-consistent estimator based on observed data.
• Figure 4: Type 1 error rate and power of a one-sample t-test at $\alpha=5\%$ for the population mean of $age$ with naive model-based and corrected standard errors (SEs).
• Figure 5: Empirical coverage of $95\%$ confidence intervals for effect of $age$ on $income$, with model-based and corrected standard error (SE).