MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation

TL;DR

MoGA tackles the prohibitive cost of self-attention on very long video sequences by introducing a learnable token router that partitions tokens into semantically coherent groups and performs intra-group attention, reducing complexity to $O(N^2/M)$. The approach integrates with existing attention kernels (e.g., FlashAttention) and combines with spatial-temporal windowing to maintain local coherence, while a group-balancing loss prevents routing collapse. Coupled with a data pipeline for minute-level, multi-shot training and a cross-modal shot-level conditioning scheme, MoGA enables end-to-end long video generation at 480p/24fps with context lengths around $580{,}000$ tokens. Empirical results show improvements over both sparse-attention baselines and multi-stage methods across short, multi-shot, and long videos, alongside substantial compute savings and emergent cross-shot consistency.

Abstract

Long video generation with Diffusion Transformers (DiTs) is bottlenecked by the quadratic scaling of full attention with sequence length. Since attention is highly redundant, outputs are dominated by a small subset of query-key pairs. Existing sparse methods rely on blockwise coarse estimation, whose accuracy-efficiency trade-offs are constrained by block size. This paper introduces Mixture-of-Groups Attention (MoGA), an efficient sparse attention that uses a lightweight, learnable token router to precisely match tokens without blockwise estimation. Through semantic-aware routing, MoGA enables effective long-range interactions. As a kernel-free method, MoGA integrates seamlessly with modern attention stacks, including FlashAttention and sequence parallelism. Building on MoGA, we develop an efficient long video generation model that end-to-end produces minute-level, multi-shot, 480p videos at 24 fps, with a context length of approximately 580k. Comprehensive experiments on various video generation tasks validate the effectiveness of our approach.

Figures (12)

• Figure 1: Illustration of our motivation. (a) Full attention suffers from dense computing when dealing with long sequences. (b) Block sparse attention lu2025moba may fail when block-level similarity is confused, resulting in unreliable attention. (c) Mixture-of-Groups attention uses a lightweight token router (i.e., a single linear layer) that assigns tokens to specialized groups, enabling groupwise attention and efficient long-context modeling.
• Figure 2: Left: Our model adopts a DiT architecture with interleaved Visual Attention and Cross-Modal Attention blocks. Visual Attention exclusively processes visual content, whereas Cross-Modal Attention enables shot-level text conditioning, instantiated via either cross-attention wan2025wan or multi-modal attention kong2024hunyuanvideoesser2024scaling. Top-right (a): Visual Attention combining MoGA with Spatial‑Temporal Group Attention for global-local consistency. Bottom‑right (b): MoGA, where a router groups tokens and performs intra-group attention, enabling long-range global interactions.
• Figure 3: Visualization of dynamic router grouping.
• Figure 4: Multi-shot long video data pipeline.
• Figure 5: Qualitative results of MoGA and other methods. We present eight representative shots, demonstrating long-content coherence, character consistency, and visual quality.