CacheQuant: Comprehensively Accelerated Diffusion Models

TL;DR

CacheQuant addresses the bottleneck of diffusion-model inference by jointly optimizing model caching and quantization in a training-free framework. It introduces Dynamic Programming Schedule to find an optimal cache plan and Decoupled Error Correction to mitigate coupled errors, enabling substantial speedups with minimal quality loss. Across UNet- and DiT-based diffusion models on diverse datasets, CacheQuant achieves notable gains in speed and compression while maintaining competitive FID/IS/CLIP metrics, outperforming traditional temporal- and structural-level acceleration methods. This approach offers a practical, hardware-friendly pathway to deploy fast diffusion generation in real-world applications.

Abstract

Diffusion models have gradually gained prominence in the field of image synthesis, showcasing remarkable generative capabilities. Nevertheless, the slow inference and complex networks, resulting from redundancy at both temporal and structural levels, hinder their low-latency applications in real-world scenarios. Current acceleration methods for diffusion models focus separately on temporal and structural levels. However, independent optimization at each level to further push the acceleration limits results in significant performance degradation. On the other hand, integrating optimizations at both levels can compound the acceleration effects. Unfortunately, we find that the optimizations at these two levels are not entirely orthogonal. Performing separate optimizations and then simply integrating them results in unsatisfactory performance. To tackle this issue, we propose CacheQuant, a novel training-free paradigm that comprehensively accelerates diffusion models by jointly optimizing model caching and quantization techniques. Specifically, we employ a dynamic programming approach to determine the optimal cache schedule, in which the properties of caching and quantization are carefully considered to minimize errors. Additionally, we propose decoupled error correction to further mitigate the coupled and accumulated errors step by step. Experimental results show that CacheQuant achieves a 5.18 speedup and 4 compression for Stable Diffusion on MS-COCO, with only a 0.02 loss in CLIP score. Our code are open-sourced: https://github.com/BienLuky/CacheQuant .

Figures (10)

• Figure 1: An overview of motivations. (a) The principles and properties of the traditional acceleration methods at each level. (b) Our approach integrates the advantages of model caching and quantization while eliminating their drawbacks, achieving comprehensive acceleration at two levels.
• Figure 2: An overview of CacheQuant. DPS selects the optimal cache schedule and DEC mitigates the coupled and accumulated errors.
• Figure 3: Performance and acceleration of different optimization strategies. EDA-DM and Deepcache are optimization methods for model quantization and caching, respectively.
• Figure 4: Output errors of network at each time step.
• Figure 5: (a) Correlations between the different out-channels of $O_g$ and $O_{cq}$. (b) Box plots visualize the mean and variance of different errors. Data comes from steps $t=192$ and $t=210$ for LDM on ImageNet, which are assigned to the same group by the DPS.