Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Should I use Synthetic Data for That? An Analysis of the Suitability of Synthetic Data for Data Sharing and Augmentation

Bogdan Kulynych, Theresa Stadler, Jean Louis Raisaro, Carmela Troncoso

TL;DR

This paper analyzes whether synthetic data is a practical solution for data access, scarcity, and under-representation by formalising three use cases: data sharing as a proxy for proprietary data, augmentation to improve machine learning performance, and augmentation or imputation to reduce variance in statistical estimation. It shows that achieving simultaneous privacy and broad validity is generally impossible for data sharing, unless the analysis family is narrowly defined and supported by specialized generation methods. For ML augmentation, success hinges on external augmentation sources containing information about the target distribution and on having representative target-domain validation data; otherwise improvements may be illusory or non-generalizable. In statistical estimation, naive augmentation leads to invalid inferences, requiring specialized procedures (e.g., prediction-powered inference) with limited applicability, underscoring that synthetic data is not a universal remedy and should be used with carefully validated guarantees and problem-specific strategies.

Abstract

Recent advances in generative modelling have led many to see synthetic data as the go-to solution for a range of problems around data access, scarcity, and under-representation. In this paper, we study three prominent use cases: (1) Sharing synthetic data as a proxy for proprietary datasets to enable statistical analyses while protecting privacy, (2) Augmenting machine learning training sets with synthetic data to improve model performance, and (3) Augmenting datasets with synthetic data to reduce variance in statistical estimation. For each use case, we formalise the problem setting and study, through formal analysis and case studies, under which conditions synthetic data can achieve its intended objectives. We identify fundamental and practical limits that constrain when synthetic data can serve as an effective solution for a particular problem. Our analysis reveals that due to these limits many existing or envisioned use cases of synthetic data are a poor problem fit. Our formalisations and classification of synthetic data use cases enable decision makers to assess whether synthetic data is a suitable approach for their specific data availability problem.

Should I use Synthetic Data for That? An Analysis of the Suitability of Synthetic Data for Data Sharing and Augmentation

TL;DR

This paper analyzes whether synthetic data is a practical solution for data access, scarcity, and under-representation by formalising three use cases: data sharing as a proxy for proprietary data, augmentation to improve machine learning performance, and augmentation or imputation to reduce variance in statistical estimation. It shows that achieving simultaneous privacy and broad validity is generally impossible for data sharing, unless the analysis family is narrowly defined and supported by specialized generation methods. For ML augmentation, success hinges on external augmentation sources containing information about the target distribution and on having representative target-domain validation data; otherwise improvements may be illusory or non-generalizable. In statistical estimation, naive augmentation leads to invalid inferences, requiring specialized procedures (e.g., prediction-powered inference) with limited applicability, underscoring that synthetic data is not a universal remedy and should be used with carefully validated guarantees and problem-specific strategies.

Abstract

Recent advances in generative modelling have led many to see synthetic data as the go-to solution for a range of problems around data access, scarcity, and under-representation. In this paper, we study three prominent use cases: (1) Sharing synthetic data as a proxy for proprietary datasets to enable statistical analyses while protecting privacy, (2) Augmenting machine learning training sets with synthetic data to improve model performance, and (3) Augmenting datasets with synthetic data to reduce variance in statistical estimation. For each use case, we formalise the problem setting and study, through formal analysis and case studies, under which conditions synthetic data can achieve its intended objectives. We identify fundamental and practical limits that constrain when synthetic data can serve as an effective solution for a particular problem. Our analysis reveals that due to these limits many existing or envisioned use cases of synthetic data are a poor problem fit. Our formalisations and classification of synthetic data use cases enable decision makers to assess whether synthetic data is a suitable approach for their specific data availability problem.
Paper Structure (19 sections, 3 equations, 3 figures)

This paper contains 19 sections, 3 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Sharing synthetic data as a proxy for replicating analyses on proprietary data
  3. Problem formalisation
  4. Synthetic data as a drop-in replacement for real data lacks statistical validity
  5. Synthetic data cannot ensure validity for too many analyses and preserve privacy at the same time
  6. Case studies
  7. Takeaways
  8. Augmenting training datasets with synthetic data
  9. Problem formalisation
  10. To improve target performance, the augmentation source must contain relevant information
  11. Circular dependency of representative validation data
  12. Case studies
  13. Takeaways
  14. Augmentation or imputation for statistical estimation
  15. Problem formalisation
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Sharing synthetic data as a proxy for replicating analyses on proprietary data: To overcome data access barriers, the data controller shares a synthetic in place of the real data with the data user who runs the desired analysis $f({\tilde{S}})$ on the synthetic instead of the real source data $S$.
  • Figure 2: Synthetic data augmentation problems can be categorised along two main dimensions. The first dimension is the source information: (A) External augmentation source vs. (B) Bootstrapped augmentation. The synthetic data ${\tilde{S}}$ used to augment the base data $S_\mathsf{base}$ can either be (A) derived from an additional augmentation source or (B) derived from the base data itself. The second dimension is the goal: (C) Augmentation to increase sample size (in-distribution) vs. (D) Augmentation for domain generalisation (out-of-distribution). The goal of data augmentation can be to improve performance on (C) in-distribution samples such as minority groups from the base domain or (D) an unknown target domain that differs from the data available to the model developer.
  • Figure 3: Decision chart for potential uses and limits of synthetic data as a solution to data availability problems.