Boosting Data Analytics With Synthetic Volume Expansion

TL;DR

The effectiveness of statistical methods on synthetic data and the privacy risks of synthetic data are explored, and the generational effect is revealed, which reveals that the error rate of statistical methods on synthetic data decreases with the addition of more synthetic data but may eventually rise or stabilize.

Abstract

Synthetic data generation, a cornerstone of Generative Artificial Intelligence, promotes a paradigm shift in data science by addressing data scarcity and privacy while enabling unprecedented performance. As synthetic data becomes more prevalent, concerns emerge regarding the accuracy of statistical methods when applied to synthetic data in contrast to raw data. This article explores the effectiveness of statistical methods on synthetic data and the privacy risks of synthetic data. Regarding effectiveness, we present the Synthetic Data Generation for Analytics framework. This framework applies statistical approaches to high-quality synthetic data produced by generative models like tabular diffusion models, which, initially trained on raw data, benefit from insights from pertinent studies through transfer learning. A key finding within this framework is the generational effect, which reveals that the error rate of statistical methods on synthetic data decreases with the addition of more synthetic data but may eventually rise or stabilize. This phenomenon, stemming from the challenge of accurately mirroring raw data distributions, highlights a "reflection point"-an ideal volume of synthetic data defined by specific error metrics. Through three case studies, sentiment analysis, predictive modeling of structured data, and inference in tabular data, we validate the superior performance of this framework compared to conventional approaches. On privacy, synthetic data imposes lower risks while supporting the differential privacy standard. These studies underscore synthetic data's untapped potential in redefining data science's landscape.

Figures (10)

• Figure 1: Illustration of denoising diffusion probabilistic model ho2020denoising. In the forward process, noise $\epsilon_t$ sequentially corrupts the sample $\bm X_t$, evolving from the original sample $\bm X_0$ to a target, such as random noise, over $t=0,\cdots, T$. Conversely, the backward process employs a neural network $\epsilon_{\theta} (\bm X_t, t)$ to predict $\epsilon_t$, starting from the random state. This network, fine-tuned from similar pre-trained models, denoises $\bm X_t$, from $t = T, ..., 0$, to generate a synthetic sample $\tilde{\bm X}_0$ replicating $\bm X_0$.
• Figure 2: Comparative error analysis of CatBoost, Syn-Boost, and FNN, with Syn-Boost and FNN applying transfer learning with the same distributions across eight benchmarks kotelnikov2023tabddpm, measured at various synthetic-to-raw data ratios. The stars indicate the size of the pre-training data used to obtain pre-trained models and the tuned sample size for Syn-Boost. The performance for classification and regression tasks is measured by misclassification rate and RMSE, respectively. Point-wise standard errors, derived from smoothing spline models hastie2009elements, are also depicted to illustrate the variation in error.
• Figure 3: Marginal distributions of datasets categorized as female, synthetic female, and male are illustrated, with legends arranged from top to bottom. Normalized bar and kernel density plots represent categorical and numerical features, respectively.
• Figure 4: Pairwise correlation plots between raw and synthetic female datasets compare those between raw female and male datasets, accompanied by their differences. Dark cells in the difference plots signify pronounced deviations from the female distribution. Pearson's correlation, Correlation Ratio, and Theil's U measure continuous-continuous, categorical-continuous, and categorical-categorical correlations.
• Figure 5: Comparative error analysis of CatBoost, Syn-Boost, and FNN, with Syn-Boost and FNN utilizing transfer learning with distinct distributions on the Adult-Female dataset adult, with Adult-Male data serving as pre-training data, across various synthetic-to-raw data ratios. Stars indicate the pre-training data size and the tuned sample size for Syn-Boost. The vertical bars, calculated using smoothing spline models hastie2009elements, represent the pointwise standard error.

Theorems & Definitions (6)

• Theorem 2.1: Reflection Point
• Theorem 2.2: Accuracy Gain
• Theorem 2.3: Validity and power of Syn-Test
• proof
• proof
• proof