Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PQD: Post-training Quantization for Efficient Diffusion Models

Jiaojiao Ye, Zhen Wang, Linnan Jiang

TL;DR

Diffusion models achieve high-fidelity image synthesis but incur heavy compute due to the multi-step forward and reverse process with variance schedule $\beta_t$; PQD proposes a training-free, time-aware post-training quantization framework to quantize full-precision models to 8-bit or 4-bit using time-step aware calibration. It uses a Gaussian time-step distribution $\mathcal{N}(\mu,\sigma)$ for calibration and applies QDrop to recover performance. It extends to latent-space quantization enabling 512×512 text-guided generation and incorporates conditioning for text prompts. Across ImageNet unconditional generation and MS-COCO 512×512 text-to-image tasks, PQD yields competitive or state-of-the-art results among PTQ methods, enabling practical deployment on resource-constrained hardware.

Abstract

Diffusionmodels(DMs)havedemonstratedremarkableachievements in synthesizing images of high fidelity and diversity. However, the extensive computational requirements and slow generative speed of diffusion models have limited their widespread adoption. In this paper, we propose a novel post-training quantization for diffusion models (PQD), which is a time-aware optimization framework for diffusion models based on post-training quantization. The proposed framework optimizes the inference process by selecting representative samples and conducting time-aware calibration. Experimental results show that our proposed method is able to directly quantize full-precision diffusion models into 8-bit or 4-bit models while maintaining comparable performance in a training-free manner, achieving a few FID change on ImageNet for unconditional image generation. Our approach demonstrates compatibility and can also be applied to 512x512 text-guided image generation for the first time.

PQD: Post-training Quantization for Efficient Diffusion Models

TL;DR

Diffusion models achieve high-fidelity image synthesis but incur heavy compute due to the multi-step forward and reverse process with variance schedule ; PQD proposes a training-free, time-aware post-training quantization framework to quantize full-precision models to 8-bit or 4-bit using time-step aware calibration. It uses a Gaussian time-step distribution for calibration and applies QDrop to recover performance. It extends to latent-space quantization enabling 512×512 text-guided generation and incorporates conditioning for text prompts. Across ImageNet unconditional generation and MS-COCO 512×512 text-to-image tasks, PQD yields competitive or state-of-the-art results among PTQ methods, enabling practical deployment on resource-constrained hardware.

Abstract

Diffusionmodels(DMs)havedemonstratedremarkableachievements in synthesizing images of high fidelity and diversity. However, the extensive computational requirements and slow generative speed of diffusion models have limited their widespread adoption. In this paper, we propose a novel post-training quantization for diffusion models (PQD), which is a time-aware optimization framework for diffusion models based on post-training quantization. The proposed framework optimizes the inference process by selecting representative samples and conducting time-aware calibration. Experimental results show that our proposed method is able to directly quantize full-precision diffusion models into 8-bit or 4-bit models while maintaining comparable performance in a training-free manner, achieving a few FID change on ImageNet for unconditional image generation. Our approach demonstrates compatibility and can also be applied to 512x512 text-guided image generation for the first time.
Paper Structure (13 sections, 3 equations, 3 figures, 1 table, 1 algorithm)

This paper contains 13 sections, 3 equations, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. Diffusion Models
  5. Post-training Quantization
  6. Exploration of PTQ on DMs
  7. Post-training Quantization for DMs (PQD)
  8. Extension to High-Resolution Image Generation
  9. Experiments
  10. Experiment Setup
  11. Unconditional Generation
  12. Text-to-image Generation
  13. Conclusion

Figures (3)

  • Figure 1: The activation of the output layer varies during the denoising process.
  • Figure 2: PQD construct calibration dataset with time step distributed over denoising process. Our method generate inputs that are accurate reflections of data seen during the production in a data-free manner.
  • Figure 3: Text-guided generated samples with 512x512 resolution by Stable Diffusion model. Upper Samples generated using the full-precision model. Middle Samples generated by our 8-bit quantized model. Bottom Samples generated by 8-bit Linear Quantization model.