Is Synthetic Image Useful for Transfer Learning? An Investigation into Data Generation, Volume, and Utilization

TL;DR

The paper addresses how synthetic images from text-to-image models can aid transfer learning, uncovering a distribution gap that harms naive data mixing. It proposes a two-stage Bridge Transfer framework—first fine-tuning on synthetic data, then adapting with real data—and introduces Dataset Style Inversion to align synthetic and real styles. Across 10 downstream datasets and multiple architectures, the approach yields consistent improvements and faster convergence, with gains up to $53\%$ on certain tasks and indications that larger synthetic volumes can further boost performance. The work offers a practical, scalable path to leverage Generative AI for transfer learning, especially in data-scarce domains, while highlighting conditions under which synthetic data are most beneficial.

Abstract

Synthetic image data generation represents a promising avenue for training deep learning models, particularly in the realm of transfer learning, where obtaining real images within a specific domain can be prohibitively expensive due to privacy and intellectual property considerations. This work delves into the generation and utilization of synthetic images derived from text-to-image generative models in facilitating transfer learning paradigms. Despite the high visual fidelity of the generated images, we observe that their naive incorporation into existing real-image datasets does not consistently enhance model performance due to the inherent distribution gap between synthetic and real images. To address this issue, we introduce a novel two-stage framework called bridged transfer, which initially employs synthetic images for fine-tuning a pre-trained model to improve its transferability and subsequently uses real data for rapid adaptation. Alongside, We propose dataset style inversion strategy to improve the stylistic alignment between synthetic and real images. Our proposed methods are evaluated across 10 different datasets and 5 distinct models, demonstrating consistent improvements, with up to 30% accuracy increase on classification tasks. Intriguingly, we note that the enhancements were not yet saturated, indicating that the benefits may further increase with an expanded volume of synthetic data.

Figures (10)

• Figure 1: We present our bridged transfer learning framework, which initially employs synthetic images for fine-tuning a pre-trained model to improve its transferability and subsequently use real images for rapid adaption (Sec. \ref{['sec:bridged_transfer']}). Under this framework, we also explore the influence of synthetic images volume (Sec. \ref{['sec_data_volume']}) and style alignment between real and synthetic images, achieved through our proposed dataset style inversion technique (Sec.\ref{['sec_data_syn']}).
• Figure 2: Test accuracy comparison across 10 target tasks using 4 different transfer pipelines.
• Figure 3: Training accuracy convergence of vanilla and bridged transfer on 5 target datasets. Bridged transfer accelerates the training convergence.
• Figure 4: Test accuracy of different transfer pipelines on various few-shot downstream datasets. The x-axis is the number of shots (real images) per class.
• Figure 5: Investigation on the number of synthetic data used for bridged transfer.