Rectified SpaAttn: Revisiting Attention Sparsity for Efficient Video Generation

TL;DR

This work tackles the high computational cost of diffusion transformer video generation by addressing the shortcomings of attention sparsity. It introduces Rectified SpaAttn, a training-free, plug-in module that aligns sparse attention with implicit full attention using IPAR to rectify critical-token biases and GAPR to manage non-critical-token gains, all guided by pooled QK statistics. The method achieves substantial end-to-end speedups (up to 3.33× at high sparsity) while maintaining or improving video quality, and it is compatible with Triton-accelerated kernels and existing caching techniques. The approach is validated across multiple video-generation backbones, with extensive ablations demonstrating the effectiveness of IPAR and GAPR and the practical value of the implicit-full-attention rationale. The work provides open-source implementations and visualization tools to facilitate adoption in real-world diffusion-video pipelines.

Abstract

Diffusion Transformers dominate video generation, but the quadratic complexity of attention computation introduces substantial latency. Attention sparsity reduces computational costs by focusing on critical tokens while ignoring non-critical tokens. However, existing methods suffer from severe performance degradation. In this paper, we revisit attention sparsity and reveal that existing methods induce systematic biases in attention allocation: (1) excessive focus on critical tokens amplifies their attention weights; (2) complete neglect of non-critical tokens causes the loss of relevant attention weights. To address these issues, we propose Rectified SpaAttn, which rectifies attention allocation with implicit full attention reference, thereby enhancing the alignment between sparse and full attention maps. Specifically: (1) for critical tokens, we show that their bias is proportional to the sparse attention weights, with the ratio governed by the amplified weights. Accordingly, we propose Isolated-Pooling Attention Reallocation, which calculates accurate rectification factors by reallocating multimodal pooled weights. (2) for non-critical tokens, recovering attention weights from the pooled query-key yields attention gains but also introduces pooling errors. Therefore, we propose Gain-Aware Pooling Rectification, which ensures that the rectified gain consistently surpasses the induced error. Moreover, we customize and integrate the Rectified SpaAttn kernel using Triton, achieving up to 3.33 and 2.08 times speedups on HunyuanVideo and Wan 2.1, respectively, while maintaining high generation quality. We release Rectified SpaAttn as open-source at https://github.com/BienLuky/Rectified-SpaAttn .

Figures (9)

• Figure 1: Rectified SpaAttn achieves significant speedup while maintaining high quality, making video generation more efficient.
• Figure 2: Sparsity results in systematic attention biases.
• Figure 3: Motivation. (a) Attention sparsity introduces allocation biases that cause large output errors, which intensify as sparsity increases. (b) Implicit full attention obtained by IPAR exhibits strong alignment with the true full attention in both magnitude and distribution, as evidenced by lower normalized L1 error and higher cosine similarity across timesteps and layers under 80% sparsity. The normalized L1 error is computed as the L1 difference normalized by the magnitude of the true attention. Data come from HunyuanVideo.
• Figure 4: Overview of Rectified SpaAttn. Sparse attention is computed at the block level, exhibiting significant misalignment with full attention. IPAR isolates text tokens during pooling and reallocates pooled weights to obtain distribution-aligned implicit full attention, rectifying biases of critical blocks. GAPR estimates rectified gains and pooling errors to generate compensation mask, applying implicit attention compensation to non-critical blocks. The rectified sparse attention achieves strong alignment with the full attention.
• Figure 5: Left: Attention maps from direct pooling. Right: Attention recall of HunyuanVideo under 80% sparsity.