LoFT: LoRA-fused Training Dataset Generation with Few-shot Guidance

TL;DR

LoFT tackles the distribution mismatch between synthetic and real data in supervised learning with diffusion models by introducing per-image LoRA adapters and a fusion mechanism at generation time. It learns a separate LoRA ΔW^(i) for each real image and fuses two such adapters via h_out = W h_in + λ ΔW^(i) h_in + (1−λ) ΔW^(j) h_in to produce high-fidelity yet diverse samples. Extensive experiments across ImageNet and nine fine-grained datasets show LoFT often outperforms zero-shot and other few-shot synthetic-data methods in CLIP fine-tuning, and can surpass real k-shot baselines with modest synthetic data; ablations confirm the effectiveness of two-LoRA fusion with λ ≈ 0.5. This approach enables scalable, high-quality synthetic dataset generation with practical impact for data-scarce domains, while highlighting trade-offs between fidelity, diversity, and computational overhead.

Abstract

Despite recent advances in text-to-image generation, using synthetically generated data seldom brings a significant boost in performance for supervised learning. Oftentimes, synthetic datasets do not faithfully recreate the data distribution of real data, i.e., they lack the fidelity or diversity needed for effective downstream model training. While previous work has employed few-shot guidance to address this issue, existing methods still fail to capture and generate features unique to specific real images. In this paper, we introduce a novel dataset generation framework named LoFT, LoRA-Fused Training-data Generation with Few-shot Guidance. Our method fine-tunes LoRA weights on individual real images and fuses them at inference time, producing synthetic images that combine the features of real images for improved diversity and fidelity of generated data. We evaluate the synthetic data produced by LoFT on 10 datasets, using 8 to 64 real images per class as guidance and scaling up to 1000 images per class. Our experiments show that training on LoFT-generated data consistently outperforms other synthetic dataset methods, significantly increasing accuracy as the dataset size increases. Additionally, our analysis demonstrates that LoFT generates datasets with high fidelity and sufficient diversity, which contribute to the performance improvement. The code is available at https://github.com/ExplainableML/LoFT.

Figures (15)

• Figure 1: LoFT: Given a few real images per class, we first adapt a diffusion model to each image using LoRA. Next, two LoRA weights corresponding to images of the same class are randomly selected and fused to generate new images. The generated synthetic images above show diverse colors and compositions while maintaining the swing object.
• Figure 2: LoFT pipeline. In the first phase, given a few real images per class, we adapt a diffusion model to each image using LoRA. In the second phase, two LoRA weights corresponding to images of the same class are randomly selected and fused to generate new synthetic images. These generated images are then compiled to form a dataset for training the classification model.
• Figure 3: Classification accuracy on ImageNet when fine-tuning CLIP on synthetic data generated from different methods at different scales. We report few-shot guidance on 8, 16, 32, and 64 images per class and a baseline of training CLIP only on k-shot real data. LoFT consistently outperforms other methods and real k-shot result with small amount of synthetic data.
• Figure 4: Per-class analysis on synthetic datasets generated from different methods. The color indicates a log-likelihood of the ImageNet validation dataset when CLIP is fine-tuned on the 0.5M-sized synthetic dataset in the 16-shot setting.
• Figure 5: Qualitative examples for the classes Acoustic guitar and Hourglass from ImageNet. Our LoFT method generates diverse images, such as variations in zoom level, for acoustic guitar, and preserves an object of interest better for hourglass.