Accelerating Diffusion-based Video Editing via Heterogeneous Caching: Beyond Full Computing at Sampled Denoising Timestep

Abstract

Diffusion-based video editing has emerged as an important paradigm for high-quality and flexible content generation. However, despite their generality and strong modeling capacity, Diffusion Transformers (DiT) remain computationally expensive due to the iterative denoising process, posing challenges for practical deployment. Existing video diffusion acceleration methods primarily exploit denoising timestep-level feature reuse, which mitigates the redundancy in denoising process, but overlooks the architectural redundancy within the DiT that many attention operations over spatio-temporal tokens are redundantly executed, offering little to no incremental contribution to the model output. This work introduces HetCache, a training-free diffusion acceleration framework designed to exploit the inherent heterogeneity in diffusion-based masked video-to-video (MV2V) generation and editing. Instead of uniformly reuse or randomly sampling tokens, HetCache assesses the contextual relevance and interaction strength among various types of tokens in designated computing steps. Guided by spatial priors, it divides the spatial-temporal tokens in DiT model into context and generative tokens, and selectively caches the context tokens that exhibit the strongest correlation and most representative semantics with generative ones. This strategy reduces redundant attention operations while maintaining editing consistency and fidelity. Experiments show that HetCache achieves a noticeable acceleration, including a 2.67$\times$ latency speedup and FLOPs reduction over commonly used foundation models, with negligible degradation in editing quality.

Figures (6)

• Figure 1: (a). Illustration of the acceleration dimensions in Diffusion Transformers (DiTs). Unlike existing methods, the proposed Heterogeneous Caching (HetCache) jointly models denoising-step redundancy in the diffusion process and token redundancy within the Transformer backbone. (b). As a tailored heterogeneous strategy, HetCache accelerates diffusion-based masked video-to-video (MV2V) editing while maintaining generation quality.
• Figure 2: The overview of our proposed HetCache scheme. In denoising process, we use the timestep-embeddings-modulated-input liu2025timestep to estimate the computing demand. According to the accumulated distance, Full-Compute anchor step, Reuse step and Partial-Compute step will be executed. In full-computing, HetCache will use spatial prior extracted from editing mask to categorize the DiT tokens into Context, Margin, and Generative Tokens. The Context Tokens which takes high portion and cause redundant computation cost will be cached for partial-compute steps according to its semantic representativeness and interaction strength with the generative tokens.
• Figure 3: VBench comparison between HetCache and other methods on different video editing tasks.
• Figure 4: Visualization of different video editing tasks. HetCache produces relatively high-quality results while other methods suffer from smoothness, ghosting, and blurring issues.
• Figure 5: Key metircs comparison of different $K$ and $r_{\text{ctx}}$ setting in context token sampling.