Qua$^2$SeDiMo: Quantifiable Quantization Sensitivity of Diffusion Models

TL;DR

Qua$^2$SeDiMo addresses the quantization bottleneck in diffusion models by introducing a graph-based, mixed-precision PTQ framework that directly links per-layer quantization choices to end-to-end performance. It represents denoisers as DAGs and uses a GNN-based explanation mechanism with ranking losses to identify operation-level and block-level sensitivity, enabling sub-$4$-bit weight quantization across diverse architectures (U-Nets and Diffusion Transformers). The method yields cost-efficient configurations (e.g., $3.4$–$3.9$-bit weights) paired with $6$-bit activations that outperform existing approaches (Q-Diffusion, TFMQ-DM, ViDiT-Q) in FID/CLIP quality, while providing interpretable insights into which layers and blocks drive performance under quantization. This leads to practical reductions in inference cost without calibration data and offers architectural guidance for quantizing future diffusion models.

Abstract

Diffusion Models (DM) have democratized AI image generation through an iterative denoising process. Quantization is a major technique to alleviate the inference cost and reduce the size of DM denoiser networks. However, as denoisers evolve from variants of convolutional U-Nets toward newer Transformer architectures, it is of growing importance to understand the quantization sensitivity of different weight layers, operations and architecture types to performance. In this work, we address this challenge with Qua$^2$SeDiMo, a mixed-precision Post-Training Quantization framework that generates explainable insights on the cost-effectiveness of various model weight quantization methods for different denoiser operation types and block structures. We leverage these insights to make high-quality mixed-precision quantization decisions for a myriad of diffusion models ranging from foundational U-Nets to state-of-the-art Transformers. As a result, Qua$^2$SeDiMo can construct 3.4-bit, 3.9-bit, 3.65-bit and 3.7-bit weight quantization on PixArt-$α$, PixArt-$Σ$, Hunyuan-DiT and SDXL, respectively. We further pair our weight-quantization configurations with 6-bit activation quantization and outperform existing approaches in terms of quantitative metrics and generative image quality.

Figures (18)

• Figure 1: Example $512^2$ images generated using PixArt-$\alpha$. We compare images from the full precision model to those generated by a quantized denoiser using different PTQ techniques. Specifically, we compare Q-Diffusion, TFMQ-DM and ViDiT-Q at W4 precision to three configurations built by Qua$^2$SeDiMo - W4, W3.7 and W3.4 - with and without 6-bit activation quantization.
• Figure 2: PixArt-$\alpha$/$\Sigma$ images at FP16 precision and quantized to W4A16 by $K$-Means, UAQ and Q-Diffusion. COCO prompt: 'A jet with smoke pouring from its wings'.
• Figure 3: Induced DiT subgraphs. Attention weights (red) are captured in a 4-hop subgraph rooted at 'Proj Out'. The feedforward module is a 1-hop subgraph rooted at 'FF 2'. Yellow box: Each weight layer can be quantized using three methods and two bit-precision levels.
• Figure 4: Results on PixArt-$\alpha$, PixArt-$\Sigma$, Hunyuan and SDXL under constrained optimization to minimize FID and $\widebar{Bits}$. Dashed horizonal line denotes the FID of the W16A16 model. Dotted grey line denotes the Pareto frontier constructed from our corpus of randomly sampled configurations (yellow dots). For each predictor ensemble, we generate two quantization configurations: 'Op-level' for individual weight layers and 'Block-level' for subgraph structures. Purple circles denote configurations we later investigate to generate images and draw insights from. Best viewed in color.
• Figure 5: PixArt-$\Sigma$ example images and comparison with related work. Resolution: $1024^2$.