Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Bias-Variance Decomposition for Ensembles over Multiple Synthetic Datasets

Ossi Räisä, Antti Honkela

TL;DR

This work provides a bias-variance decomposition for generative ensembles that train predictors on multiple independently generated synthetic datasets. It shows the MSE (and BS) breaks into interpretable components, with a clear 1/m scaling of variance terms, and delivers a practical rule-of-thumb for the optimal number of synthetic datasets. The authors extend the framework to differentially private generators and non-i.i.d. settings, and validate the theory across diverse datasets, showing that multiple synthetic datasets typically yield meaningful accuracy gains, especially for high-variance predictors, while a single large synthetic dataset offers limited improvement. The results offer actionable guidance for designing synthetic-data pipelines and have implications for model evaluation and DP data sharing strategies.

Abstract

Recent studies have highlighted the benefits of generating multiple synthetic datasets for supervised learning, from increased accuracy to more effective model selection and uncertainty estimation. These benefits have clear empirical support, but the theoretical understanding of them is currently very light. We seek to increase the theoretical understanding by deriving bias-variance decompositions for several settings of using multiple synthetic datasets, including differentially private synthetic data. Our theory yields a simple rule of thumb to select the appropriate number of synthetic datasets in the case of mean-squared error and Brier score. We investigate how our theory works in practice with several real datasets, downstream predictors and error metrics. As our theory predicts, multiple synthetic datasets often improve accuracy, while a single large synthetic dataset gives at best minimal improvement, showing that our insights are practically relevant.

A Bias-Variance Decomposition for Ensembles over Multiple Synthetic Datasets

TL;DR

This work provides a bias-variance decomposition for generative ensembles that train predictors on multiple independently generated synthetic datasets. It shows the MSE (and BS) breaks into interpretable components, with a clear 1/m scaling of variance terms, and delivers a practical rule-of-thumb for the optimal number of synthetic datasets. The authors extend the framework to differentially private generators and non-i.i.d. settings, and validate the theory across diverse datasets, showing that multiple synthetic datasets typically yield meaningful accuracy gains, especially for high-variance predictors, while a single large synthetic dataset offers limited improvement. The results offer actionable guidance for designing synthetic-data pipelines and have implications for model evaluation and DP data sharing strategies.

Abstract

Recent studies have highlighted the benefits of generating multiple synthetic datasets for supervised learning, from increased accuracy to more effective model selection and uncertainty estimation. These benefits have clear empirical support, but the theoretical understanding of them is currently very light. We seek to increase the theoretical understanding by deriving bias-variance decompositions for several settings of using multiple synthetic datasets, including differentially private synthetic data. Our theory yields a simple rule of thumb to select the appropriate number of synthetic datasets in the case of mean-squared error and Brier score. We investigate how our theory works in practice with several real datasets, downstream predictors and error metrics. As our theory predicts, multiple synthetic datasets often improve accuracy, while a single large synthetic dataset gives at best minimal improvement, showing that our insights are practically relevant.
Paper Structure (39 sections, 9 theorems, 59 equations, 16 figures, 13 tables)

This paper contains 39 sections, 9 theorems, 59 equations, 16 figures, 13 tables.

Table of Contents

  1. INTRODUCTION
  2. Related Work
  3. BIAS-VARIANCE DECOMPOSITIONS FOR GENERATIVE ENSEMBLES
  4. Problem Setting
  5. Mean-squared Error Decomposition
  6. Practical Considerations
  7. Estimating the effect of Multiple Synthetic Datasets
  8. Differentially Private Synthetic Data Generators
  9. EXPERIMENTS
  10. Effect of Multiple Synthetic Datasets
  11. Predicting Performance from Two Synthetic Datasets
  12. Differentially Private Synthetic Data
  13. DISCUSSION
  14. MISSING PROOFS
  15. NON-I.I.D. SYNTHETIC DATA
  16. ...and 24 more sections

Key Result

Theorem 2.1

Let the parameters for $m$ generators $\theta_i \sim p(\theta | D_r)$, $i=1, \dots, m$, be i.i.d. Let the synthetic datasets be $D_s^{i} \sim p(D_s | \theta_i)$ independently, and let $\hat{g}(x; D_s^{1:m}) = \frac{1}{m}\sum_{i=1}^m g(x; D_s^i)$. Then the mean-squared error in predicting $y$ from $x where $f(x) = \operatorname*{\mathbb{E}}_{y}[y]$ is the optimal predictor for real data, $\theta \

Figures (16)

  • Figure 1: MSE on regression datasets (a) or Brier score on classification datasets (b) of the ensemble of downstream predictors, with varying number of synthetic datasets $m$ from synthpop. Increasing the number of synthetic datasets generally decreases both metrics, especially for decision trees and 1-NN. The predictors are nearest neighbours with 1 or 5 neighbours (1-NN and 5-NN), decision tree (DT), random forest (RF), a multilayer perceptron (MLP), gradient boosted trees (GB), a support vector machine (SVM), ridge regression (RR) and logistic regression (LogR). The black line is the MSE of the best predictor on real data. Tables \ref{['table:abalone-results']} to \ref{['table:german-credit-brier-results']} in the Appendix contain the numbers from the plots.
  • Figure 2: Estimating the MV and SDV terms from the decomposition. Decision trees have high variances on all datasets, while linear, ridge and logistic regression have low variances. MV depends mostly on the predictor, while SDV depends on both the predictor and synthetic data generation algorithm. The points are the averages of estimated MV and SDV, averaged over the test data, from 3 repeats with different train-test splits. See Figure \ref{['fig:variance-estimation']} in the Appendix for results on all datasets.
  • Figure 3: MSE or Brier score prediction on the ACS 2018 dataset. The predictions are very accurate on this dataset. The solid lines for DDPM and synthpop (SP-P) show the same error MSE or Brier score as Figure \ref{['fig:main-results']}, while the dashed lines show predicted MSE or Brier score. 1-NN and 5-NN are nearest neighbours with 1 or 5 neighbours. We omitted downstream algorithms with uninteresting flat curves. See Figure \ref{['fig:mse-prediction-regression1']} in the Appendix for the full figure, and Figures \ref{['fig:mse-prediction-regression2']} and \ref{['fig:mse-prediction-classification1']} for the other datasets. Tables \ref{['table:mse-prediction-acs-2018']} and \ref{['table:mse-prediction-german-credit']} in the Appendix contain the numbers from the plots.
  • Figure 4: Brier score of the ensemble of downstream predictors with varying numbers of synthetic datasets $m$, generated with the DP methods AIM or NAPSU-MQ from the Adult dataset with a reduced set of features. Increasing the number of datasets generally decreases the score, even with AIM, which splits the privacy budget between $m$ synthetic datasets. The privacy parameters are $\epsilon = 1.5$, $\delta = n^{-2} \approx 4.7\cdot 10^{-7}$. The predictors are the same as in Figure \ref{['fig:main-results']}. The black lines show the loss of the best non-DP downstream predictor trained on real data. Table \ref{['table:dp-experiment-brier']} contains the numbers from the plots, and Figure \ref{['fig:dp-experiment-all-metrics']} contains plots of the other error metrics.
  • Figure S1: Comparison of synthetic data generation algorithms for several prediction algorithms on the California housing dataset, with 1 to 10 synthetic datasets. DDPM and synthpop achieve smaller MSE in the downstream predictions, so they were selected for further experiments. SP-P and SP-IP are the proper and improper variants of synthpop, and DDPM-KL is DDPM with KL divergence loss. 1-NN and 5-NN are nearest neighbours with 1 and 5 neighbours. The dashed black lines show the performance of each prediction algorithm on the real data, and the solid black line shows the performance of the best predictor, random forest, on the real data. The results are averaged over 3 repeats, with different train-test splits. The error bars are 95% confidence intervals formed by bootstrapping over the repeats. Linear regression was omitted, as it had nearly identical results as ridge regression. Table \ref{['table:synthetic-data-algo-comparison']} in the Appendix contains the numbers in the plots, including ridge regression.
  • ...and 11 more figures

Theorems & Definitions (15)

  • Theorem 2.1
  • Corollary 2.2
  • proof
  • Theorem 2.3
  • Theorem A.1
  • proof
  • Theorem A.1
  • proof
  • Theorem B.1
  • proof
  • ...and 5 more