SmoothCache: A Universal Inference Acceleration Technique for Diffusion Transformers

TL;DR

This work introduces SmoothCache, a training-free, model-agnostic caching technique for Diffusion Transformer (DiT) inference that exploits cross-timestep layer similarity. By calibrating a single global threshold $\alpha$ using a small set of samples, SmoothCache caches layer outputs before residual connections and reuses them when the average layer-variation falls below $\alpha$, reducing compute with minimal quality loss. The method is demonstrated across image, video, and audio diffusion tasks, achieving up to $71\%$ speedups while maintaining or improving key generation metrics and showing compatibility with multiple solvers. Overall, SmoothCache provides a practical, multimodal acceleration approach that can enable real-time diffusion applications without task-specific retraining.

Abstract

Diffusion Transformers (DiT) have emerged as powerful generative models for various tasks, including image, video, and speech synthesis. However, their inference process remains computationally expensive due to the repeated evaluation of resource-intensive attention and feed-forward modules. To address this, we introduce SmoothCache, a model-agnostic inference acceleration technique for DiT architectures. SmoothCache leverages the observed high similarity between layer outputs across adjacent diffusion timesteps. By analyzing layer-wise representation errors from a small calibration set, SmoothCache adaptively caches and reuses key features during inference. Our experiments demonstrate that SmoothCache achieves 8% to 71% speed up while maintaining or even improving generation quality across diverse modalities. We showcase its effectiveness on DiT-XL for image generation, Open-Sora for text-to-video, and Stable Audio Open for text-to-audio, highlighting its potential to enable real-time applications and broaden the accessibility of powerful DiT models.

Figures (8)

• Figure 1: Accelerating Diffusion Transformer inference across multiple modalities with 50 DDIM Steps on DiT-XL-256x256, 100 DPMSolver++(3M) SDE steps for a 10s audio sample (spectrogram shown) on Stable Audio Open, 30 Rectified Flow steps on Open-Sora 480p 2s videos.
• Figure 2: L1 Relative Error Curves of different architecture components. Curves are plotted with 95% confidence intervals from 10 calibration samples from all components explored in this paper and scaled to the same y-axis range. Note that OpenSora has distinct spatial and temporal diffusion blocks.
• Figure 3: SmoothCache-Eligible Layers of candidate models. This visualization highlights the targeted layers that precede residual connections in a DiT block for each architecture. Each model contains $N$ DiT blocks. In the original DiT-XL model, Self-attention and Feed-forward layers are cached. In the Stable Audio Open model, Self-attention, Cross-attention, and Feed-forward layers are cached. In the Open Sora model, Self-attention, Cross-attention, and Feed-forward layers across both the temporal and spatial partitions of the DiT block.
• Figure 4: Layer Compute Composition of candidate models. These are computed from the MACs of the default model configurations without SmoothCache applied. Note that in all candidate models, SmoothCache eligible layers comprise at least 90% of compute time.
• Figure 5: SmoothCache results on DiT-XL/2-256$\times$256 for unconditional generation with 50 DDIM sampling steps on ImageNet-1k for thresholds 0.08 and 0.18, as well as for Static Caching.