Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximation

Dong Liu, Yanxuan Yu, Jiayi Zhang, Yifan Li, Ben Lengerich, Ying Nian Wu

Abstract

Diffusion Transformers (DiT) are powerful generative models but remain computationally intensive due to their iterative structure and deep transformer stacks. To alleviate this inefficiency, we propose \textbf{FastCache}, a hidden-state-level caching and compression framework that accelerates DiT inference by exploiting redundancy within the model's internal representations. FastCache introduces a dual strategy: (1) a spatial-aware token selection mechanism that adaptively filters redundant tokens based on hidden-state saliency, and (2) a transformer-level cache that reuses latent activations across timesteps when changes fall below a predefined threshold. These modules work jointly to reduce unnecessary computation while preserving generation fidelity through learnable linear approximation. Theoretical analysis shows that FastCache maintains bounded approximation error under a hypothesis-testing-based decision rule. Empirical evaluations across multiple DiT variants demonstrate substantial reductions in latency and memory usage, achieving the best generation quality among existing cache methods, as measured by FID and t-FID. To further improve the speedup of FastCache, we also introduce a token merging module that merges redundant tokens based on k-NN density. Code is available at \href{https://github.com/NoakLiu/FastCache-xDiT}{https://github.com/NoakLiu/FastCache-xDiT}.

FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximation

Abstract

Diffusion Transformers (DiT) are powerful generative models but remain computationally intensive due to their iterative structure and deep transformer stacks. To alleviate this inefficiency, we propose \textbf{FastCache}, a hidden-state-level caching and compression framework that accelerates DiT inference by exploiting redundancy within the model's internal representations. FastCache introduces a dual strategy: (1) a spatial-aware token selection mechanism that adaptively filters redundant tokens based on hidden-state saliency, and (2) a transformer-level cache that reuses latent activations across timesteps when changes fall below a predefined threshold. These modules work jointly to reduce unnecessary computation while preserving generation fidelity through learnable linear approximation. Theoretical analysis shows that FastCache maintains bounded approximation error under a hypothesis-testing-based decision rule. Empirical evaluations across multiple DiT variants demonstrate substantial reductions in latency and memory usage, achieving the best generation quality among existing cache methods, as measured by FID and t-FID. To further improve the speedup of FastCache, we also introduce a token merging module that merges redundant tokens based on k-NN density. Code is available at \href{https://github.com/NoakLiu/FastCache-xDiT}{https://github.com/NoakLiu/FastCache-xDiT}.

Paper Structure

This paper contains 49 sections, 7 theorems, 49 equations, 4 figures, 15 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Overview
  5. Spatial-Temporal Token Reduction Module
  6. Transformer-Level Caching Module
  7. Spatial-Temporal Token Merging
  8. Interpretability of FastCache: Learnable Linear Approximation for Motion-Background Separation
  9. Preliminaries: Hidden States and Redundancy
  10. Interaction-Theoretic Interpretation
  11. Theoretical Proposition: First-order Interaction Recovery
  12. Interpretability Insight: Symbolic Decomposition via Caching
  13. Experiments
  14. Experimental Setup
  15. Implementation Details
  16. ...and 34 more sections

Key Result

Proposition 1

Let $v : \mathbb{R}^{N \times D} \rightarrow \mathbb{R}$ be a thrice-differentiable function over hidden states. Suppose: Then, the first-order Taylor approximation: is equivalent to recovering first-order interactions:

Figures (4)

  • Figure 1: Top: A video exhibiting substantial motion, with the corresponding masked input displayed to the right. The masked regions indicate areas excluded from DiT processing. Middle: An illustration of the hidden-state interaction space, where the colormap visualizes the magnitude of first-order derivatives. Warmer colors (e.g., red) represent higher absolute gradient values, while cooler colors (e.g., blue) indicate lower values. Bottom: A comparatively static video, with its masked input shown on the right. Interpretation: Videos characterized by greater frame-to-frame variation, such as the top example, contain a higher proportion of motion-related tokens. These tokens are associated with the red regions in the heatmap and tend to exhibit a lower cache utilization rate (motion region needs recomputation), these tokens are associated with the blue regions in the heatmap and tend to exhibit a higher cache utilization rate (static background will be cached by FastCache for acceleration)
  • Figure 2: Left: Illustration of the Spatial-Temporal Token Reduction Module. Right: Illustration of the Transformer-Level Caching Module.
  • Figure 3: Impact of statistical threshold $\alpha$ on caching rate and FID.
  • Figure 4: FastCache Image Generation after compression; FastCache preserves image structure and motion fidelity with fewer computation steps.

Theorems & Definitions (13)

  • Proposition 1
  • proof
  • Theorem 1: Cache Approximation under Drift
  • proof
  • Theorem 2: Generalization of Caching Approximation
  • proof
  • Lemma 1: Shapley Linearization Error
  • Theorem 3: Taylor-Harsanyi Equivalence for FastCache
  • proof
  • Theorem 4: Robustness to n-th Order Residuals
  • ...and 3 more