Efficient Autoregressive Video Diffusion with Dummy Head

TL;DR

This work identifies a subset of attention heads in autoregressive video diffusion models that underutilize historical frames, coining them as dummy heads. It introduces Dummy Forcing, a training-free framework comprising heterogeneous memory allocation (HMA), dynamic head programming (DHP) via a greedy dynamic-programming solution, and packed attention forward (PAF) to aggressively prune dummy-head caches while preserving context-critical information. Across classic, high-resolution, and long-context video tasks, the method delivers up to 2x end-to-end speedups with minimal quality degradation, including real-time 24 FPS performance on 5-second videos and substantial gains on 1080P generation. The approach is compatible with existing acceleration techniques and demonstrates strong generalization to multiple autoregressive diffusion models, highlighting practical impact for scalable, efficient video generation without additional training.

Abstract

The autoregressive video diffusion model has recently gained considerable research interest due to its causal modeling and iterative denoising. In this work, we identify that the multi-head self-attention in these models under-utilizes historical frames: approximately 25% heads attend almost exclusively to the current frame, and discarding their KV caches incurs only minor performance degradation. Building upon this, we propose Dummy Forcing, a simple yet effective method to control context accessibility across different heads. Specifically, the proposed heterogeneous memory allocation reduces head-wise context redundancy, accompanied by dynamic head programming to adaptively classify head types. Moreover, we develop a context packing technique to achieve more aggressive cache compression. Without additional training, our Dummy Forcing delivers up to 2.0x speedup over the baseline, supporting video generation at 24.3 FPS with less than 0.5% quality drop. Project page is available at https://csguoh.github.io/project/DummyForcing/.

Figures (22)

• Figure 1: Weak context utilization in the multi-head attention of existing methods, e.g., Diffusion Forcing chen2024diffusionforcing, Self Forcing huang2025selfforcing. Naively pruning all KV caches of 25% heads results in only a marginal performance drop (84.0 vs. 83.78) while speedup inference from 17.6FPS to 19.6FPS.
• Figure 2: The proposed Dummy Forcing can be applied to (1) efficiently generate videos, (2) overcome quadratic complexity in high-resolution video generation, and (3) enlarge context lengths without increasing computational overhead.
• Figure 3: We compute the frame attention score by summing across rows and averaging across columns within the sink/neighbor/current frame group.
• Figure 4: (a)-(c): We gather attention maps from all heads and use \ref{['eq:attn_cal']} to compute the frame attention scores on the sink/neighbor/current frames. (d): The core set ratio under different conditions. For each bar, we change the corresponding condition while keeping the others fixed. We provide more implementation details of the observation experiment in the Appendix.
• Figure 5: (a) We assign different contextual receptive fields to different head types. (b) A toy example of classifying different head types, with num_head=8 and $N$=4 in this case. (c) We fuse different heads by context packing for more aggressive compression.