Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sparse-to-Sparse Training of Diffusion Models

Inês Cardoso Oliveira, Decebal Constantin Mocanu, Luis A. Leiva

TL;DR

Diffusion models are powerful but computationally demanding; this paper introduces sparse-to-sparse training for diffusion models, enabling training from scratch with fixed (Static-DM) or cycling (RigL-DM, MagRan-DM) sparsity. By applying SST and DST to Latent Diffusion and ChiroDiff across six datasets, the authors show that sparse DMs can match or surpass dense counterparts while reducing parameters and FLOPs, with dynamic sparsity (25–50% sparsity) often yielding the best results and a conservative prune/regrowth ratio (p≈0.05) performing well at high sparsity. The study provides practical guidelines on sparsity levels and regrowth strategies, highlighting substantial memory and compute savings for both training and inference, and pointing to future hardware and software developments needed to fully exploit sparse DM benefits. Overall, sparse-to-sparse training offers a promising route to more efficient diffusion models without sacrificing sample quality, enabling broader accessibility and reduced environmental impact.

Abstract

Diffusion models (DMs) are a powerful type of generative models that have achieved state-of-the-art results in various image synthesis tasks and have shown potential in other domains, such as natural language processing and temporal data modeling. Despite their stable training dynamics and ability to produce diverse high-quality samples, DMs are notorious for requiring significant computational resources, both in the training and inference stages. Previous work has focused mostly on increasing the efficiency of model inference. This paper introduces, for the first time, the paradigm of sparse-to-sparse training to DMs, with the aim of improving both training and inference efficiency. We focus on unconditional generation and train sparse DMs from scratch (Latent Diffusion and ChiroDiff) on six datasets using three different methods (Static-DM, RigL-DM, and MagRan-DM) to study the effect of sparsity in model performance. Our experiments show that sparse DMs are able to match and often outperform their Dense counterparts, while substantially reducing the number of trainable parameters and FLOPs. We also identify safe and effective values to perform sparse-to-sparse training of DMs.

Sparse-to-Sparse Training of Diffusion Models

TL;DR

Diffusion models are powerful but computationally demanding; this paper introduces sparse-to-sparse training for diffusion models, enabling training from scratch with fixed (Static-DM) or cycling (RigL-DM, MagRan-DM) sparsity. By applying SST and DST to Latent Diffusion and ChiroDiff across six datasets, the authors show that sparse DMs can match or surpass dense counterparts while reducing parameters and FLOPs, with dynamic sparsity (25–50% sparsity) often yielding the best results and a conservative prune/regrowth ratio (p≈0.05) performing well at high sparsity. The study provides practical guidelines on sparsity levels and regrowth strategies, highlighting substantial memory and compute savings for both training and inference, and pointing to future hardware and software developments needed to fully exploit sparse DM benefits. Overall, sparse-to-sparse training offers a promising route to more efficient diffusion models without sacrificing sample quality, enabling broader accessibility and reduced environmental impact.

Abstract

Diffusion models (DMs) are a powerful type of generative models that have achieved state-of-the-art results in various image synthesis tasks and have shown potential in other domains, such as natural language processing and temporal data modeling. Despite their stable training dynamics and ability to produce diverse high-quality samples, DMs are notorious for requiring significant computational resources, both in the training and inference stages. Previous work has focused mostly on increasing the efficiency of model inference. This paper introduces, for the first time, the paradigm of sparse-to-sparse training to DMs, with the aim of improving both training and inference efficiency. We focus on unconditional generation and train sparse DMs from scratch (Latent Diffusion and ChiroDiff) on six datasets using three different methods (Static-DM, RigL-DM, and MagRan-DM) to study the effect of sparsity in model performance. Our experiments show that sparse DMs are able to match and often outperform their Dense counterparts, while substantially reducing the number of trainable parameters and FLOPs. We also identify safe and effective values to perform sparse-to-sparse training of DMs.
Paper Structure (37 sections, 2 equations, 13 figures, 8 tables, 2 algorithms)

This paper contains 37 sections, 2 equations, 13 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Diffusion Models
  4. Efficiency in Diffusion Models
  5. Sparse-to-Sparse Training
  6. Static Sparse Training.
  7. Dynamic Sparse Training.
  8. Methodology
  9. Static Sparse Training: Static-DM
  10. Dynamic Sparse Training: MagRan-DM and RigL-DM
  11. Experimental Setup
  12. Models and Benchmarks
  13. Latent Diffusion.
  14. ChiroDiff.
  15. Experimental Details
  16. ...and 22 more sections

Figures (13)

  • Figure 1: Sparsification of Latent Diffusion (\ref{['fig:diagLD']}) and ChiroDiff (\ref{['fig:diagCD']}) models.
  • Figure 2: FID score comparisons between Dense, and Static-DM, MagRan-DM and RigL-DM with various sparsity levels, for Latent Diffusion, with prune and regrowth ratio $p=0.5$. Values are averaged over 3 runs.
  • Figure 3: FID score comparisons between Dense, and Static-DM, MagRan-DM and RigL-DM with various sparsity levels, for ChiroDiff, with prune and regrowth rate $p=0.5$. Values averaged over 3 runs.
  • Figure 4: FID score comparisons between Dense, and Static-DM, MagRan-DM and RigL-DM with $S=0.75$, using varied diffusion timesteps for Latent Diffusion (CelebA-HQ), and ChiroDiff (KanjiVG). Values averaged over 3 runs.
  • Figure 5: FID scores comparison between Dense and DST models with various prune and regrowth ratios, for Latent Diffusion on CelebA-HQ.
  • ...and 8 more figures