Forecast the Principal, Stabilize the Residual: Subspace-Aware Feature Caching for Efficient Diffusion Transformers

TL;DR

Diffusion transformers incur high computational cost during iterative denoising. We introduce SVD-Cache, a subspace-aware feature caching framework that decomposes diffusion features into a dominant principal subspace and an orthogonal residual, predicting the principal component with exponential moving average and reusing the residual. A one-time SVD offline on a reference prompt yields a universal subspace basis enabling fast reconstruction of the low-rank component from cached components. The final feature combines the EMA-predicted principal part with the reused residual. Experiments show robust, near-lossless speedups on image and video diffusion tasks and compatibility with quantization, distillation, and sparse attention, offering a practical plug-and-play path to accelerate diffusion transformer inference.

Abstract

Diffusion Transformer (DiT) models have achieved unprecedented quality in image and video generation, yet their iterative sampling process remains computationally prohibitive. To accelerate inference, feature caching methods have emerged by reusing intermediate representations across timesteps. However, existing caching approaches treat all feature components uniformly. We reveal that DiT feature spaces contain distinct principal and residual subspaces with divergent temporal behavior: the principal subspace evolves smoothly and predictably, while the residual subspace exhibits volatile, low-energy oscillations that resist accurate prediction. Building on this insight, we propose SVD-Cache, a subspace-aware caching framework that decomposes diffusion features via Singular Value Decomposition (SVD), applies exponential moving average (EMA) prediction to the dominant low-rank components, and directly reuses the residual subspace. Extensive experiments demonstrate that SVD-Cache achieves near-lossless across diverse models and methods, including 5.55$\times$ speedup on FLUX and HunyuanVideo, and compatibility with model acceleration techniques including distillation, quantization and sparse attention. Our code is in supplementary material and will be released on Github.

Figures (6)

• Figure 1: (a) PCA visualization of different feature spaces: Original Full Feature Subspace shows oscillatory trajectory while Principal Subspace evolves smoothly. Residual subspace is oscillatory and low-energy. (b) Singular values and right singular vectors are stable across different prompts. Similarity is evaluated using the product of cosine similarity and magnitude similarity. A similarity over 0.8 typically indicates a stable and reusable subspace.
• Figure 2: Overview of the proposed SVD-Cache framework. At each diffusion timestep, we decompose the feature map into principal and residual subspaces via SVD. The principal subspace, which captures the most significant variations and evolves smoothly, is predicted using exponential moving average (EMA). The residual subspace, containing high-frequency details that resist accurate prediction, is directly reused. This hybrid strategy allows us to efficiently cache features while maintaining high fidelity in the generated outputs.
• Figure 3: Visual Comparison of 5.5 $\times$ accelerated FLUX between different feature cache methods.
• Figure 4: Comparison between SVD-Cache and TaylorSeer on HunyuanVideo. SVD-Cache sustains stronger temporal consistency and visual quality under more aggressive acceleration, while TaylorSeer develops visible artifacts and flickering.
• Figure 5: Overall and Ablation Results of SVD-Cache. (a) SVD-Cache consistently outperforms token-wise(ToCa) and other full-feature-space predictor baselines. (b) When the low-rank subspace is predicted with other ODE methods, quality is significantly improved over the original method. (c) Ablation study on Flux by applying different strategies to the low-rank and residual components separately. (d) Ablation study on the energy threshold $\tau$ for subspace decomposition.