Real-Fake: Effective Training Data Synthesis Through Distribution Matching

TL;DR

This paper reframes training data synthesis as a distribution-matching problem, aiming to align both the data distribution $q(x)$ with $p_{\theta}(x)$ and the conditional $q(y|x)$ with $p_{\theta}(y|x)$ to train high-performance classifiers without over-reliance on real data. It introduces a diffusion-model-based framework enhanced with Maximum Mean Discrepancy (MMD)–driven objectives, text-vision conditioned generation, and informative latent-prior initialization, instantiated in a Latent Diffusion Model (Stable Diffusion) with LoRAFinetuning. Empirically, synthetic data alone reaches $70.9\%$ top-1 on ImageNet1K with $1\times$ real data and $76.0\%$ with $10\times$, while augmentation and data-scale strategies improve both in-distribution and out-of-distribution performance and offer privacy advantages (lower membership-inference risk and non-memorizing visuals). The findings demonstrate that principled distribution matching can close much of the gap between synthetic and real data, with practical implications for scalable, privacy-conscious data synthesis in large-scale vision tasks.

Abstract

Synthetic training data has gained prominence in numerous learning tasks and scenarios, offering advantages such as dataset augmentation, generalization evaluation, and privacy preservation. Despite these benefits, the efficiency of synthetic data generated by current methodologies remains inferior when training advanced deep models exclusively, limiting its practical utility. To address this challenge, we analyze the principles underlying training data synthesis for supervised learning and elucidate a principled theoretical framework from the distribution-matching perspective that explicates the mechanisms governing synthesis efficacy. Through extensive experiments, we demonstrate the effectiveness of our synthetic data across diverse image classification tasks, both as a replacement for and augmentation to real datasets, while also benefits such as out-of-distribution generalization, privacy preservation, and scalability. Specifically, we achieve 70.9% top1 classification accuracy on ImageNet1K when training solely with synthetic data equivalent to 1 X the original real data size, which increases to 76.0% when scaling up to 10 X synthetic data.

Figures (4)

• Figure 1: Left: Visualization of the synthetic and real ImageNet data distribution using the first two principal components of features extracted by the CLIP image encoder. Our synthetic data better aligns with real data distribution than the baseline (vanilla Stable Diffusion). Middle: Our synthetic data achieves better performance compared with the baseline, and can effectively augment real data across all datasets. Right: Scaling up synthetic training data can improve the image classification performances in both in-distribution and out-of-distribution (OOD) tasks, even outperform training with real data in OOD tasks.
• Figure 2: Effect of Scaling Up Synthetic Dataset Top-1 image classification performance with synthetic data only (indicated by blue solid curve) increases with synthetic dataset size, eventually outperforming real data (indicated by red dash line). The horizontal axis represents the amount of synthetic data used as multiples of the original real dataset size.
• Figure 3: Membership Inference Attack (MIA) Performance with LiRA. LiRA achieves a TPR of 0.001% at a low FPR of 0.1% when applied to synthetic data, while the results for private data is 0.01%, which indicates training with synthetic data is much more privacy-preserving.
• Figure 4: Visualization on synthetic and retrieved real data with SSCD. The synthesized data does not exhibit evident copying or memorization.