Towards Accurate Post-training Quantization for Diffusion Models

TL;DR

This work tackles data-free post-training quantization for diffusion models by addressing two key weaknesses of prior PTQ methods: cross-timestep activation distribution variance and uninformative calibration data. It introduces distribution-aware, group-wise activation quantization across timesteps, with a differentiable search to assign timesteps to groups and learn group-specific rounding parameters, plus an SRM-based strategy to select informative calibration timesteps. The approach combines a differentiable objective that couples discretization error with entropy-regularized group weights, and a timestep selection criterion that balances empirical risk and distributional similarity via MMD. Empirical results show substantial gains over state-of-the-art data-free PTQ methods on both unconditional and conditioned diffusion models, at 6-bit or 8-bit, with similar computational cost, enabling efficient deployment on resource-constrained devices.

Abstract

In this paper, we propose an accurate data-free post-training quantization framework of diffusion models (ADP-DM) for efficient image generation. Conventional data-free quantization methods learn shared quantization functions for tensor discretization regardless of the generation timesteps, while the activation distribution differs significantly across various timesteps. The calibration images are acquired in random timesteps which fail to provide sufficient information for generalizable quantization function learning. Both issues cause sizable quantization errors with obvious image generation performance degradation. On the contrary, we design group-wise quantization functions for activation discretization in different timesteps and sample the optimal timestep for informative calibration image generation, so that our quantized diffusion model can reduce the discretization errors with negligible computational overhead. Specifically, we partition the timesteps according to the importance weights of quantization functions in different groups, which are optimized by differentiable search algorithms. We also select the optimal timestep for calibration image generation by structural risk minimizing principle in order to enhance the generalization ability in the deployment of quantized diffusion model. Extensive experimental results show that our method outperforms the state-of-the-art post-training quantization of diffusion model by a sizable margin with similar computational cost.

Figures (8)

• Figure 1: (a) Existing methods leverage shared quantization for activation discretization across different timesteps with significant quantization errors, while we divide timesteps into groups with specific rounding functions for each partition. (b) Conventional methods construct calibration set by randomly image selecting with ineffective supervision, while we actively sample timesteps based on the structural risk minimization (SRM) principle.
• Figure 2: The overall pipeline of our method. The calibration images are generated according to the selected timesteps, and activations in the pre-trained diffusion models are parallelly quantized by rounding functions of all groups. The output feature maps are acquired by adding the quantized value with the importance weights, where the quantization parameters and the importance weights are jointly optimized. The importance weight entropy and the sampling times are considered in the timestep selection criteria to decide the optimal timestep for calibration image generation in the next round.
• Figure 3: (a) The evolution of branch importance weights during the differentiable search. (b) The generation quality w.r.t. different hyperparameters $\lambda$ and $\eta$.
• Figure 4: The images generated by quantized Stable Diffusion models and the corresponding text prompts, where different post-training quantization methods are employed.
• Figure 5: $256\times256$ LSUN-Church samples from 100 step LDMs in 6-bit with different post-training quantization methods.