An Analysis on Quantizing Diffusion Transformers

TL;DR

Transformer-only diffusion models are powerful but resource-intensive. The authors propose a retraining-free post-training quantization pipeline that combines 1-step activation calibration with group-wise weight quantization to achieve low-bit quantization. They demonstrate improved SQNR and competitive image generation quality on conditional tasks without optimizations, highlighting practicality for hardware deployment. This work provides a baseline for non-training-based PTQ of diffusion models and guides future efforts toward efficient diffusion transformers.

Abstract

Diffusion Models (DMs) utilize an iterative denoising process to transform random noise into synthetic data. Initally proposed with a UNet structure, DMs excel at producing images that are virtually indistinguishable with or without conditioned text prompts. Later transformer-only structure is composed with DMs to achieve better performance. Though Latent Diffusion Models (LDMs) reduce the computational requirement by denoising in a latent space, it is extremely expensive to inference images for any operating devices due to the shear volume of parameters and feature sizes. Post Training Quantization (PTQ) offers an immediate remedy for a smaller storage size and more memory-efficient computation during inferencing. Prior works address PTQ of DMs on UNet structures have addressed the challenges in calibrating parameters for both activations and weights via moderate optimization. In this work, we pioneer an efficient PTQ on transformer-only structure without any optimization. By analysing challenges in quantizing activations and weights for diffusion transformers, we propose a single-step sampling calibration on activations and adapt group-wise quantization on weights for low-bit quantization. We demonstrate the efficiency and effectiveness of proposed methods with preliminary experiments on conditional image generation.

Figures (4)

• Figure 1: Without optimizations, data ranges pose challenges to quantize both activations and weights especially at a lower bit-width. (Above) Activation range varies dynamically across sampling steps and significant outliers persist. (Below) Weights are quantized channel-wise, but dispearsed outliers for each channel introduces high quantization loss when compressed to a lower bit.
• Figure 2: The quantization error is measured in $\mathbf{SQNR}$. There is a discrepancy between the first and the last sampling process. The calibrated paramters are most robust when the added noise is the strongest.
• Figure 3: Calibrating through 50 steps produces visible image noise. 1-step calibration generates quality closer to the full precision output. Outputs from conditional DiT 8A8W.
• Figure 4: Qualitative examples of two quantization settings: 8A8W and 8A4W. Proposed improvements restore the original image quality.