Mixture of Distributions Matters: Dynamic Sparse Attention for Efficient Video Diffusion Transformers

TL;DR

MOD-DiT addresses the $O(N^2)$ bottleneck of self-attention in video diffusion transformers by introducing a sampling-free dynamic sparse attention framework. It identifies three evolving attention patterns—block-diagonal, parallel-to-main-diagonal, and vertical—and models their time-varying intensities with a linear approximation, enabling real-time mask prediction without sampling. The method reconstructs complete attention maps, predicts pattern intensities across denoising steps, and generates dynamic masks, all aided by hardware-accelerated least-squares computation. Empirical results across CogVideoX-v1.5, HunyuanVideo, and Wan2.1 show consistent speedups (up to about 2.3×) with preserved or improved video quality, demonstrating practical impact for scalable video diffusion generation. The approach is training-free and broadly applicable to existing vDiTs, offering a flexible tradeoff between efficiency and accuracy for real-world deployment.

Abstract

While Diffusion Transformers (DiTs) have achieved notable progress in video generation, this long-sequence generation task remains constrained by the quadratic complexity inherent to self-attention mechanisms, creating significant barriers to practical deployment. Although sparse attention methods attempt to address this challenge, existing approaches either rely on oversimplified static patterns or require computationally expensive sampling operations to achieve dynamic sparsity, resulting in inaccurate pattern predictions and degraded generation quality. To overcome these limitations, we propose a \underline{\textbf{M}}ixtrue-\underline{\textbf{O}}f-\underline{\textbf{D}}istribution \textbf{DiT} (\textbf{MOD-DiT}), a novel sampling-free dynamic attention framework that accurately models evolving attention patterns through a two-stage process. First, MOD-DiT leverages prior information from early denoising steps and adopts a {distributed mixing approach} to model an efficient linear approximation model, which is then used to predict mask patterns for a specific denoising interval. Second, an online block masking strategy dynamically applies these predicted masks while maintaining historical sparsity information, eliminating the need for repetitive sampling operations. Extensive evaluations demonstrate consistent acceleration and quality improvements across multiple benchmarks and model architectures, validating MOD-DiT's effectiveness for efficient, high-quality video generation while overcoming the computational limitations of traditional sparse attention approaches.

Figures (14)

• Figure 1: Comparison of the visualization effects of different sparse attention methods on HunyuanVideokong2024hunyuanvideo. Our method MOD-DiT consistently achieves 2.2$\times$ speedup, and keep almost the same as original videos.
• Figure 2: Visualization of the four attention patterns in CogVideoX-v1.5yang2024cogvideox.
• Figure 3: Evolution of the Attention Sparsity Map in CogVideoX-v1.5yang2024cogvideox (Layer 0, Head 9) over Denoising Steps, which demonstrates significant differences in the attention sparsity patterns across denoising steps.
• Figure 4: Normalized approximation error of linear approximation model (eq. \ref{['eq:motivation_approx_model']}) across sequence lengths on hunyuan video.
• Figure 5: Evolution of vertical and parallel-diagonal pattern intensities across denoising steps, showing convergence to piecewise linearity.

Theorems & Definitions (3)

• Remark 1
• Remark 2
• Remark 3