Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VMonarch: Efficient Video Diffusion Transformers with Structured Attention

Cheng Liang, Haoxian Chen, Liang Hou, Qi Fan, Gangshan Wu, Xin Tao, Limin Wang

TL;DR

VMonarch tackles the attention bottleneck in Video Diffusion Transformers by representing sparse spatio-temporal attention with Monarch matrices and optimizing them via alternating minimization. It couples a spatio-temporal Monarch factorization with a First-Frame Recomputation strategy and a novel Online-Entropy FlashAttention kernel to enable fast updates for long video sequences while preserving generation quality. Empirically, VMonarch achieves comparable or superior quality to full attention on VBench, while delivering up to $17.5\times$ reduction in attention FLOPs and over $5\times$ kernel speedups for long videos, outperforming state-of-the-art sparse methods at $90\%$ sparsity. The approach generalizes across model scales and longer sequence lengths, offering a practical path toward scalable diffusion-based video generation.

Abstract

The quadratic complexity of the attention mechanism severely limits the context scalability of Video Diffusion Transformers (DiTs). We find that the highly sparse spatio-temporal attention patterns exhibited in Video DiTs can be naturally represented by the Monarch matrix. It is a class of structured matrices with flexible sparsity, enabling sub-quadratic attention via an alternating minimization algorithm. Accordingly, we propose VMonarch, a novel attention mechanism for Video DiTs that enables efficient computation over the dynamic sparse patterns with structured Monarch matrices. First, we adapt spatio-temporal Monarch factorization to explicitly capture the intra-frame and inter-frame correlations of the video data. Second, we introduce a recomputation strategy to mitigate artifacts arising from instabilities during alternating minimization of Monarch matrices. Third, we propose a novel online entropy algorithm fused into FlashAttention, enabling fast Monarch matrix updates for long sequences. Extensive experiments demonstrate that VMonarch achieves comparable or superior generation quality to full attention on VBench after minimal tuning. It overcomes the attention bottleneck in Video DiTs, reduces attention FLOPs by a factor of 17.5, and achieves a speedup of over 5x in attention computation for long videos, surpassing state-of-the-art sparse attention methods at 90% sparsity.

VMonarch: Efficient Video Diffusion Transformers with Structured Attention

TL;DR

VMonarch tackles the attention bottleneck in Video Diffusion Transformers by representing sparse spatio-temporal attention with Monarch matrices and optimizing them via alternating minimization. It couples a spatio-temporal Monarch factorization with a First-Frame Recomputation strategy and a novel Online-Entropy FlashAttention kernel to enable fast updates for long video sequences while preserving generation quality. Empirically, VMonarch achieves comparable or superior quality to full attention on VBench, while delivering up to reduction in attention FLOPs and over kernel speedups for long videos, outperforming state-of-the-art sparse methods at sparsity. The approach generalizes across model scales and longer sequence lengths, offering a practical path toward scalable diffusion-based video generation.

Abstract

The quadratic complexity of the attention mechanism severely limits the context scalability of Video Diffusion Transformers (DiTs). We find that the highly sparse spatio-temporal attention patterns exhibited in Video DiTs can be naturally represented by the Monarch matrix. It is a class of structured matrices with flexible sparsity, enabling sub-quadratic attention via an alternating minimization algorithm. Accordingly, we propose VMonarch, a novel attention mechanism for Video DiTs that enables efficient computation over the dynamic sparse patterns with structured Monarch matrices. First, we adapt spatio-temporal Monarch factorization to explicitly capture the intra-frame and inter-frame correlations of the video data. Second, we introduce a recomputation strategy to mitigate artifacts arising from instabilities during alternating minimization of Monarch matrices. Third, we propose a novel online entropy algorithm fused into FlashAttention, enabling fast Monarch matrix updates for long sequences. Extensive experiments demonstrate that VMonarch achieves comparable or superior generation quality to full attention on VBench after minimal tuning. It overcomes the attention bottleneck in Video DiTs, reduces attention FLOPs by a factor of 17.5, and achieves a speedup of over 5x in attention computation for long videos, surpassing state-of-the-art sparse attention methods at 90% sparsity.
Paper Structure (31 sections, 11 equations, 13 figures, 5 tables, 2 algorithms)

This paper contains 31 sections, 11 equations, 13 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Diffusion Model Inference Acceleration.
  4. Efficient Attention Mechanisms.
  5. Monarch Matrix.
  6. Method
  7. Preliminaries
  8. Video Monarch Attention
  9. Complexity Analysis.
  10. First-Frame Recomputation Strategy
  11. Online Entropy Based FlashAttention
  12. Experiments
  13. Implementation Setup
  14. Method Effectiveness
  15. Method Efficiency
  16. ...and 16 more sections

Figures (13)

  • Figure 1: Video Monarch Attention (VMonarch) leverages dynamically updated spatial-temporal Monarch matrices dao2022monarch to model sparse attention patterns in video DiTs. When generating a 321-frame video at 448$\times$832 resolution (approximately 118K tokens), VMonarch speeds up self-attention computation by 17.5$\times$ compared to full attention (FA2) dao2023flashattention and by 2$\times$ compared to highly sparse alternatives like Video Sparse Attention (VSA) at 90% sparsity. After minimal fine-tuning, VMonarch attains comparable video generation quality to FA2.
  • Figure 2: Overview of our Video Monarch Attention. (a) We represent the $N \times N$ full attention matrix via two alternatingly optimized smaller Monarch factors $\bm L$ and $\bm R$ with spatial-temporal structure factorization. (b) We introduce a Recomputation Strategy to address quality degradation caused by excessively large temperature term $\bm{c}_{\bm{R}}$ for the first frame. (c) We propose an Online-Entropy algorithm integrated with FlashAttention to accelerate the iterative computation of Monarch matrices further.
  • Figure 3: Illustration of the attention sink phenomenon in video models. The first frame tends to accumulate excessive attention from subsequent frames, leading to a loss of fine-grained details when using Monarch Attention. Our First-Frame Recomputation Strategy effectively restores these details by recalculating the attention for the first frame using full attention.
  • Figure 4: Qualitative samples of our models. We compare the generation quality of the tuned models between full attention dao2023flashattention, Video Sparse Attention (VSA) zhang2025vsa, and our VMonarch Attention.
  • Figure 5: The kernel speedup ratio over Flash Attention 2 (FA2) dao2023flashattention under varying sequence lengths. Our VMonarch demonstrates superior efficiency, achieving over 2$\times$ speedup at 28k tokens and over 5$\times$ speedup at 62k tokens compared to FA2, surpassing other sparse attention methods at $90\%$ sparsity.
  • ...and 8 more figures