ReHyAt: Recurrent Hybrid Attention for Video Diffusion Transformers

TL;DR

ReHyAt addresses the scalability bottleneck of quadratic attention in video diffusion transformers by introducing a recurrent chunked hybrid attention that preserves local softmax fidelity while applying linear attention globally. The method reformulates as a chunk-wise RNN with constant memory, enabling long-duration and on-device video generation. A lightweight two-stage training pipeline—attention distillation from a bidirectional softmax teacher followed by small-scale finetuning—yields near state-of-the-art quality at roughly 160 GPU-hours. Empirical results on VBench and VBench-2.0, plus human studies, show strong quality with substantially reduced compute and memory, unlocking practical deployment for long videos.

Abstract

Recent advances in video diffusion models have shifted towards transformer-based architectures, achieving state-of-the-art video generation but at the cost of quadratic attention complexity, which severely limits scalability for longer sequences. We introduce ReHyAt, a Recurrent Hybrid Attention mechanism that combines the fidelity of softmax attention with the efficiency of linear attention, enabling chunk-wise recurrent reformulation and constant memory usage. Unlike the concurrent linear-only SANA Video, ReHyAt's hybrid design allows efficient distillation from existing softmax-based models, reducing the training cost by two orders of magnitude to ~160 GPU hours, while being competitive in the quality. Our light-weight distillation and finetuning pipeline provides a recipe that can be applied to future state-of-the-art bidirectional softmax-based models. Experiments on VBench and VBench-2.0, as well as a human preference study, demonstrate that ReHyAt achieves state-of-the-art video quality while reducing attention cost from quadratic to linear, unlocking practical scalability for long-duration and on-device video generation. Project page is available at https://qualcomm-ai-research.github.io/rehyat.

Figures (24)

• Figure 1: A comparison of our proposed Recurrent Hybrid Attention model with Wan2.1 bidirectional full softmax attention. Top: Compute complexity increase with video duration growth (left: FLOPs, right: phone latency). Bottom: comparing our hybrid model (20$\times$ ReHyAt blocks) with original Wan2.1 1.3B, qualitatively and quantitatively. Prompt: "A cat drinking water."
• Figure 2: Overview of the temporally chunked hybrid attention arrangement without (top) and with chunk overlap (bottom).
• Figure 3: Qualitative comparison of Wan2.1 1.3B (Top) to ReHyAt 15$\times T_c$=3 (bottom) for two sample VBench prompts, "A cat and a dog." and "A dog drinking water."
• Figure 4: Comparison of attention compute (FLOPs) on 21$\times$30$\times$52 latent size (5 seconds)
• Figure 5: The total DiT FLOPs percentages versus the VBench score of original Wan2.1 1.3B model compared to various hybrid configurations or 320$\times$480 (top) and 480$\times$832 (bottom) resolutions.