SQ-DM: Accelerating Diffusion Models with Aggressive Quantization and Temporal Sparsity

TL;DR

This work tackles the slow inference of diffusion models by combining aggressive $4$-bit quantization with activation sparsity, addressing temporal error accumulation and nonuniform activation distributions. It introduces a hardware-software co-design: a heterogeneous mixed-precision dense-sparse diffusion-model accelerator with channel-last memory mapping and a time-step-aware sparsity detector. The approach demonstrates high-quality generation using 4-bit quantization (including a ReLU-based variant) and attains a total of 6.91x speed-up with 51.5% energy savings over dense baselines, highlighting strong practical potential for efficient diffusion-model deployment. These results point toward scalable, energy-efficient diffusion-model inference and set the stage for extending to video and other generative domains.

Abstract

Diffusion models have gained significant popularity in image generation tasks. However, generating high-quality content remains notably slow because it requires running model inference over many time steps. To accelerate these models, we propose to aggressively quantize both weights and activations, while simultaneously promoting significant activation sparsity. We further observe that the stated sparsity pattern varies among different channels and evolves across time steps. To support this quantization and sparsity scheme, we present a novel diffusion model accelerator featuring a heterogeneous mixed-precision dense-sparse architecture, channel-last address mapping, and a time-step-aware sparsity detector for efficient handling of the sparsity pattern. Our 4-bit quantization technique demonstrates superior generation quality compared to existing 4-bit methods. Our custom accelerator achieves 6.91x speed-up and 51.5% energy reduction compared to traditional dense accelerators.

Figures (12)

• Figure 1: Generated images and achieved speed-up for different data formats and quantization techniques.
• Figure 2: Execution process and model architecture of EDM karras2022elucidatingkarras2023analyzing.
• Figure 3: Block-wise quantization sensitivity for EDM model.
• Figure 4: EDM model computation and memory breakdown.
• Figure 5: Comparison of activation data distributions at the output of Conv+SiLU versus Conv+ReLU.