Beyond Real Data: Synthetic Data through the Lens of Regularization

TL;DR

This work tackles the problem of balancing synthetic and real data in low-data regimes by developing a learning-theoretic framework based on algorithmic stability and Wasserstein distance to bound generalization error. It shows there exists an optimal synthetic-to-real ratio that minimizes test error, predicting a nonmonotonic, U-shaped performance as synthetic data proportion changes. The analysis is first illustrated in kernel ridge regression and then extended to broader settings, including domain adaptation where synthetic target data can mitigate domain shift. Empirical validation on CIFAR-10 and real brain MRI data aligns with the theory, and the results yield practical guidelines for applying synthetic data augmentations in both in-domain and out-of-domain scenarios.

Abstract

Synthetic data can improve generalization when real data is scarce, but excessive reliance may introduce distributional mismatches that degrade performance. In this paper, we present a learning-theoretic framework to quantify the trade-off between synthetic and real data. Our approach leverages algorithmic stability to derive generalization error bounds, characterizing the optimal synthetic-to-real data ratio that minimizes expected test error as a function of the Wasserstein distance between the real and synthetic distributions. We motivate our framework in the setting of kernel ridge regression with mixed data, offering a detailed analysis that may be of independent interest. Our theory predicts the existence of an optimal ratio, leading to a U-shaped behavior of test error with respect to the proportion of synthetic data. Empirically, we validate this prediction on CIFAR-10 and a clinical brain MRI dataset. Our theory extends to the important scenario of domain adaptation, showing that carefully blending synthetic target data with limited source data can mitigate domain shift and enhance generalization. We conclude with practical guidance for applying our results to both in-domain and out-of-domain scenarios.

Figures (11)

• Figure 1: (a) Comparison of the true function $f_\star$ (blue), the synthetic generator $g$ (green), and the learned estimator $f_N$ (orange), obtained via \ref{['lem:kernel-reg']}, with parameters $r = 2.0$, $s = 0.8$, and $s' = 1.5$. (b) Prediction error $|f_N - f_\star|_{L_2}$ as a function of the regularization strength $\lambda$. The U-shaped curve attains its minimum at $\lambda^\star$ (orange dashed line), which closely matches the theoretical optimum (star marker).
• Figure 2: (a) Validation loss decreases consistently as more real data is added (blue line), while increasing synthetic data (orange dashed line) produces a U-shaped curve, indicating an optimal mixing ratio $\lambda$, as predicted by \ref{['thm:gen-gap-mixed']}. (b) Effect of distributional distance: varying the diffusion model timestep $T \in {0, 50, 150, 300}$ controls the noise level of synthetic samples. The U-shaped trend persists across all $T$ but becomes sharper with increased discrepancy between real and synthetic distributions.
• Figure 3: (a) Effect of synthetic data from distributions close to (green dashed) or far from (red dashed) the target, compared to target (blue) and source (orange) baselines. Results show the trade-off between distributional shift and regularization predicted by \ref{['thm:gen-gap-out-domain']}. (b) FID as a proxy for distributional shift: $T = 0$ (green) aligns with the target, while noisy (red) and source (orange) data show higher FID and reduced utility.
• Figure 4: Effect of synthetic-to-real data ratio and distributional distance(s) on the error rate: (a) in-domain scenario across various signal-to-noise ratios for the real dataset; (b) out-of-domain scenario. In both cases, one should ideally choose $\lambda$ such that it lies within the blue regions, which correspond to lower error rates.
• Figure 5: (a) Comparison of the true function $f_\star$ (blue), the synthetic generator $g$ (green), and the estimated function $f_N$ (orange), obtained via \ref{['lem:kernel-reg']}, with parameters $r = 2.0$, $s = 0.8$, and $s' = 0.8$. Since $g = f_\star$ in this setting, the RKHS distance is zero and all curves coincide. (b) Prediction error $\|f_N - f_\star\|_{L_2}$ as a function of the regularization strength $\lambda$. As expected, there is no U-shaped behaviour since the generator fully matches the true distribution. The theoretical optimum selects a large $\lambda$ (star marker), while the empirical optimum (dashed orange line) selects a smaller value due to numerical precision limits.

Theorems & Definitions (27)

• Lemma 2.1
• Definition 2.1: Bias-Variance Decomposition
• Theorem 2.2: Generalization Error Bound
• Corollary 2.2.1: Optimal Regularization and Synthetic Sample Size
• Definition 3.1: Uniform Stability
• Theorem 3.1: Mixed-data Generalization Bound
• Theorem 5.1: Generalization under Domain Shift
• Theorem 5.2: Mixed-data Generalization under Domain Shift
• Definition C.1: Lipschitz Continuity
• Definition C.2: Generalization Gap