Spava: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention

TL;DR

Spava tackles the efficiency challenge of long-video inference in large multimodal models by introducing a sequence-parallelism-aware approximate attention framework that distributes computation and communication across multiple GPUs. It combines frame-level parallelism, context splitting with a ZigZag virtual-host scheme, and local KV compression via passing blocks to maintain long-range dependencies without embedding compression. System-level optimizations—visual load balancing, fused forward passes, load-balanced approximate attention, and overlapped communication—drive large speedups while preserving task performance, achieving up to 12.72x speedups over strong baselines and near-lossless accuracy on several models and benchmarks. The approach demonstrates practical impact for scalable long-video understanding in real-world settings, enabling longer and more complex videos to be processed efficiently on multi-GPU clusters.

Abstract

The efficiency of long-video inference remains a critical bottleneck, mainly due to the dense computation in the prefill stage of Large Multimodal Models (LMMs). Existing methods either compress visual embeddings or apply sparse attention on a single GPU, yielding limited acceleration or degraded performance and restricting LMMs from handling longer, more complex videos. To overcome these issues, we propose Spava, a sequence-parallel framework with optimized attention that accelerates long-video inference across multiple GPUs. By distributing approximate attention, Spava reduces computation and increases parallelism, enabling efficient processing of more visual embeddings without compression and thereby improving task performance. System-level optimizations, such as load balancing and fused forward passes, further unleash the potential of Spava, delivering speedups of 12.72x, 1.70x, and 1.18x over FlashAttn, ZigZagRing, and APB, without notable performance loss. Code available at https://github.com/thunlp/APB

Figures (8)

• Figure 1: Spava's performance and inference speed using Qwen2.5VL-3B as the base LMM on VNBench (processing 64-frame 1440p videos).
• Figure 2: The framework of Spava. The anchor block and passing block are denoted as $\mathbf{B}_a$ and $\mathbf{B}_p$, while $\mathbf{B}^{(h)}$ denotes the context block on virtual host $h$. $\mathbf{B}_{qr}$ represents the query block. The video input is first encoded into embeddings using frame parallelism across hosts. After context splitting, each physical host (containing two virtual hosts) holds the anchor block, query block, and corresponding context blocks. In approximate attention, each context block is first compressed then communicated. Attention is then computed over $\mathbf{B}_a$, $\mathbf{B}^{(h)}$, $\mathbf{B}^{(2H-1-h)}$, $\mathbf{B}_p^{(h)}$, $\mathbf{B}_p^{(2H-1-h)}$, and $\mathbf{B}_{qr}$. The passing blocks are discarded immediately after attention.
• Figure 3: Fused context and query forward pass.
• Figure 4: Attention load balancing. "$|\mathbf{V}_c^{(h)}|$" is the total length of previous essential KVs of virtual host $h$.
• Figure 5: Overlapping communication with computation on virtual host $h$. "$\mathbf{Q}_{qr}\mathbf{K}^{(h_1,h_2)\top}$" estimates KV importance; "$\mathbf{B}_{c}^{(h_1,h_2)}$" are essential KVs; "query" and "merge" denote query attention and its merging; "$\mathbf{B}_a$" and "$\mathbf{B}^{(h_1,h_2)}$" indicate anchor and context attention.