Difficulty Controlled Diffusion Model for Synthesizing Effective Training Data

TL;DR

This work addresses the shortcoming of domain-aligned diffusion-based training data synthesis, which tends to generate mostly easy samples that offer limited performance gains. It introduces a difficulty-conditioned diffusion framework by adding a difficulty encoder and a fine-tuning pipeline (LoRA) that jointly model sample difficulty and maintain domain alignment. By sampling a continuous difficulty score $s$ and using class-specific prompts, the method can generate data with controlled learning difficulty, leading to more efficient performance gains (e.g., achieving competitive ImageNet results with only 10% additional synthetic data and substantial GPU-hour savings). The approach also enables visualization of hard factors, providing insights into what makes samples difficult and supporting dataset analysis across tasks and prompts.

Abstract

Generative models have become a powerful tool for synthesizing training data in computer vision tasks. Current approaches solely focus on aligning generated images with the target dataset distribution. As a result, they capture only the common features in the real dataset and mostly generate 'easy samples', which are already well learned by models trained on real data. In contrast, those rare 'hard samples', with atypical features but crucial for enhancing performance, cannot be effectively generated. Consequently, these approaches must synthesize large volumes of data to yield appreciable performance gains, yet the improvement remains limited. To overcome this limitation, we present a novel method that can learn to control the learning difficulty of samples during generation while also achieving domain alignment. Thus, it can efficiently generate valuable 'hard samples' that yield significant performance improvements for target tasks. This is achieved by incorporating learning difficulty as an additional conditioning signal in generative models, together with a designed encoder structure and training-generation strategy. Experimental results across multiple datasets show that our method can achieve higher performance with lower generation cost. Specifically, we obtain the best performance with only 10% additional synthetic data, saving 63.4 GPU hours of generation time compared to the previous SOTA on ImageNet. Moreover, our method provides insightful visualizations of category-specific hard factors, serving as a tool for analyzing datasets.

Figures (9)

• Figure 1: Our method generates images with controllable learning difficulty that align with specified difficulty score prompts. The score on each image is computed by a pretrained classifier. Additionally, our method reveals and visualizes the factors that contribute to sample difficulty.
• Figure 2: KDE distribution curve of difficulty scores. Simply fine-tuning on the whole target dataset biases the model toward generating easy images.
• Figure 3: Overview of our method. Left: Real training images are annotated with a text caption and a difficulty score assessed by a pretrained classifier. Right: A difficulty encoder is integrated into the text-to-image diffusion model. The model is fine-tuned to incorporate the difficulty score as an additional condition.
• Figure 4: Top-1 classification accuracy (%) on various tasks with ResNet-50 model. The x-axis denotes the ratio of additional synthetic images. All results are averaged over three runs, and shaded regions represent the standard deviation. The detailed numerical results are provided in the appendix.
• Figure 5: Visualization of synthetic images from four classes in Imagenette, with different difficulty score inputs shown on the left. Easy and hard samples from the target dataset are also shown for comparison. The numbers on the images are difficulty scores assessed by a pretrained ResNet-50 model.