Accelerating Diffusion Model Training under Minimal Budgets: A Condensation-Based Perspective

Abstract

Diffusion models have achieved remarkable performance on a wide range of generative tasks, yet training them from scratch is notoriously resource-intensive, typically requiring millions of training images and many GPU days. Motivated by a data-centric view of this bottleneck, we adopt a condensation-based perspective: given a large training set, the goal is to construct a much smaller condensed dataset that still supports training strong diffusion models under minimal data and compute budgets. To operationalize this perspective, we introduce Diffusion Dataset Condensation (D2C), a two-phase framework comprising Select and Attach. In the Select phase, a diffusion difficulty score combined with interval sampling is used to identify a compact, informative training subset from the original data. Building on this subset, the Attach phase further strengthens the conditional signals by augmenting each selected image with rich semantic and visual representations. To our knowledge, D2C is the first framework that systematically investigates dataset condensation for diffusion models, whereas prior condensation methods have mainly targeted discriminative architectures. Extensive experiments across data budgets (0.8%-8% of ImageNet), model architectures, and image resolutions demonstrate that D2C dramatically accelerates diffusion model training while preserving high generative quality. On ImageNet 256x256 with SiT-XL/2, D2C attains an FID of 4.3 in just 40k steps using only 0.8% of the training images, corresponding to about 233x and 100x faster training than vanilla SiT-XL/2 and SiT-XL/2 + REPA, respectively.

Figures (21)

• Figure 1: D$^2$C framework significantly accelerates diffusion model training with limited data.(a) Overview of our D$^2$C pipeline, which consists of a Select phase that filters a compact and diverse subset via diffusion difficulty score and interval sampling, and an Attach phase that enriches samples with semantic and visual information. (b)D$^2$C achieves over 100$\times$ faster convergence compared to REPA and over 233$\times$ faster than vanilla SiT-XL/2, reaching a FID of 4.3 at just 40k steps. (c) Under a strict 4% data budget (0.05M), our method achieves a FID of 2.7 at 180k iterations, demonstrating its strong training efficiency and rapid convergence.
• Figure 2: Overview of Diffusion Dataset Condensation (D$^2$C). D2C employs a two-stage process: Select and Attach. The Select stage identifies a compact and diverse subset by interval sampling using the diffusion difficulty score derived from a pre-trained diffusion model. The Attach stage further enriches each selected sample by adding semantic information and visual information.
• Figure 3: Left: Distribution of diffusion difficulty scores under different interval values $k$. Smaller intervals (e.g., 1, 2) favor low-loss samples, while larger intervals (e.g., 64, 128) result in a distribution closer to random sampling, thus approximating the original data distribution. Moderate intervals (e.g., 16) provide balanced coverage across difficulty levels. Right: Representative samples selected by three strategies: Min (lowest score), Max (highest score), and Interval (our proposed strategy). Interval sampling achieves a balance between structural clarity and contextual richness.
• Figure 4: Overview of DC-Embedding.
• Figure 5: D$^2$C improves visual quality under tight data budgets. We compare Random sampling and D$^2$C on DiT-L/2 at 10k and 50k data budgets, and neither setting uses classifier-free guidance.