DataDream: Few-shot Guided Dataset Generation

TL;DR

DataDream tackles the challenge of improving image classification with limited real data by fine-tuning a diffusion model using LoRA on few-shot examples to produce in-distribution synthetic data. It introduces two adaptation schemes, DataDream$_{dset}$ and DataDream$_{cls}$, and trains a CLIP-based classifier on synthetic (and optionally real) data, achieving state-of-the-art performance across 10 datasets in the synthetic-only setting and strong gains in the synthetic+real setting. The approach demonstrates that aligning the generative distribution to the real data via targeted, parameter-efficient fine-tuning yields more informative synthetic samples, validated by per-class distribution alignment analyses and ablation studies. Overall, DataDream offers a practical and scalable path for data-efficient training in few-shot regimes, with code available for replication and extension to other domains.

Abstract

While text-to-image diffusion models have been shown to achieve state-of-the-art results in image synthesis, they have yet to prove their effectiveness in downstream applications. Previous work has proposed to generate data for image classifier training given limited real data access. However, these methods struggle to generate in-distribution images or depict fine-grained features, thereby hindering the generalization of classification models trained on synthetic datasets. We propose DataDream, a framework for synthesizing classification datasets that more faithfully represents the real data distribution when guided by few-shot examples of the target classes. DataDream fine-tunes LoRA weights for the image generation model on the few real images before generating the training data using the adapted model. We then fine-tune LoRA weights for CLIP using the synthetic data to improve downstream image classification over previous approaches on a large variety of datasets. We demonstrate the efficacy of DataDream through extensive experiments, surpassing state-of-the-art classification accuracy with few-shot data across 7 out of 10 datasets, while being competitive on the other 3. Additionally, we provide insights into the impact of various factors, such as the number of real-shot and generated images as well as the fine-tuning compute on model performance. The code is available at https://github.com/ExplainableML/DataDream.

Figures (9)

• Figure 1: Synthetic images comparison. The previous methods for synthesizing training data sometimes misunderstand the class name due to its ambiguity (FakeIt fakeit confuses the clothes iron with the metal iron) or fail to capture fine-grained features (DISEF disef generated images lack the propeller in front of the wings in the DHC-3-800 aircraft, a red circle indicates the propeller). Meanwhile, our method accurately generates images of the class of interest and captures fine-grained details.
• Figure 2: Overview of DataDream. We fine-tune LoRA weights for the linear weights of the attention layers in both the text-encoder and the diffusion U-net to generate images closer to the few-shot images. We can train one set of DataDream weights for the whole dataset sharing common dataset-specific characteristics between classes, or a separate set of weights for each class to better learn fine-grained details of each classes.
• Figure 3: Qualitative results with increasing number of shots vs 16-shot images generated with SOTA of the class Spitfire from the FGVC Aircraft aircraft dataset. The real few-shot images at the top are used to generate the presented synthetic images at the bottom. We always use a fixed set of 16 samples, i.e. 1-shot image is a subset of 16-shots, to insure fairness in comparing results with the increasing number of shots.
• Figure 4: Distribution of FID scores per-class. The FID score is calculated per-class to measure how close the synthetic data distribution is to the real data distribution.
• Figure 5: Ablation study of DataDream. Left: We vary the number of synthetic images per class to understand the scaling effect. Right: We vary the number of real examples used for training DataDream.