Accelerating Diffusion Transformer via Increment-Calibrated Caching with Channel-Aware Singular Value Decomposition

TL;DR

This work tackles the computational bottleneck of diffusion transformers by introducing increment-calibrated caching, a training-free acceleration that leverages low-rank calibration of cached activations during inference. The core idea is to correct cached outputs with an increment term learned from the pre-trained model via SVD-based low-rank approximations, further strengthened by channel-aware scaling to mitigate outlier activations. The method yields consistent gains over naive caching across DiT-XL/2 and PixArt-$\alpha$, achieving substantial compute reductions (e.g., >45% MAC savings) while improving image quality metrics such as IS and FID, and it approaches or surpasses certain fast-sampling baselines under comparable compute budgets. The channel-aware variants (CA-SVD, CD-SVD) improve robustness by prioritizing sensitive channels, offering practical, training-free acceleration for high-fidelity image synthesis with diffusion transformers.

Abstract

Diffusion transformer (DiT) models have achieved remarkable success in image generation, thanks for their exceptional generative capabilities and scalability. Nonetheless, the iterative nature of diffusion models (DMs) results in high computation complexity, posing challenges for deployment. Although existing cache-based acceleration methods try to utilize the inherent temporal similarity to skip redundant computations of DiT, the lack of correction may induce potential quality degradation. In this paper, we propose increment-calibrated caching, a training-free method for DiT acceleration, where the calibration parameters are generated from the pre-trained model itself with low-rank approximation. To deal with the possible correction failure arising from outlier activations, we introduce channel-aware Singular Value Decomposition (SVD), which further strengthens the calibration effect. Experimental results show that our method always achieve better performance than existing naive caching methods with a similar computation resource budget. When compared with 35-step DDIM, our method eliminates more than 45% computation and improves IS by 12 at the cost of less than 0.06 FID increase. Code is available at https://github.com/ccccczzy/icc.

Figures (10)

• Figure 1: Visualization of increment-calibrated caching on DiT-XL/2.
• Figure 2: Degraded generation results of directly performing SVD on DiT-XL/2 at the resolution of 256 $\times$ 256.
• Figure 3: Overview of (a) naive caching and (b) proposed increment-calibrated caching. The former stores intermediate results of previous denoising steps and directly reuse in later timesteps which may induce unavoidable error. The proposed increment-calibrated caching corrects the cached value with calibration parameters approximated from model itself.
• Figure 4: Outlier issues of DiT models appearing in (a) input channel of FFN FC1, (b) output channel of FFN FC2, (c) input channel, and (d) output channel of MHSA output projection.
• Figure 5: The trade-off between the number of MACs and (a) IS, (b) FID, and (c) sFID for class-conditional synthesis on ImageNet with DiT-XL/2. For increment-calibrated caching, the period $p$ is set to 2.