LinVideo: A Post-Training Framework towards O(n) Attention in Efficient Video Generation

TL;DR

LinVideo tackles the quadratic cost of self-attention in video diffusion models by a data-free post-training framework that progressively replaces a target number of quadratic attention layers with linear attention. It introduces selective transfer to automatically choose which layers to linearize and ADM to align sample distributions across all timesteps along the sampling path, preserving model performance without requiring curated datasets. Empirical results show 1.43–1.71× end-to-end latency speedups on 1.3B to 14B models, with up to 15.9–20.9× speedups when combined with few-step distillation, while maintaining or surpassing baselines on key video-vision metrics. This approach enables efficient, scalable video generation in practical settings, reducing compute while retaining high-quality outputs.

Abstract

Video diffusion models (DMs) have enabled high-quality video synthesis. However, their computation costs scale quadratically with sequence length because self-attention has quadratic complexity. While linear attention lowers the cost, fully replacing quadratic attention requires expensive pretraining due to the limited expressiveness of linear attention and the complexity of spatiotemporal modeling in video generation. In this paper, we present LinVideo, an efficient data-free post-training framework that replaces a target number of self-attention modules with linear attention while preserving the original model's performance. First, we observe a significant disparity in the replaceability of different layers. Instead of manual or heuristic choices, we frame layer selection as a binary classification problem and propose selective transfer, which automatically and progressively converts layers to linear attention with minimal performance impact. Additionally, to overcome the ineffectiveness and inefficiency of existing objectives for this transfer process, we introduce an anytime distribution matching (ADM) objective that aligns the distributions of samples across any timestep along the sampling trajectory. This objective is efficient and recovers model performance. Extensive experiments show that our method achieves a 1.25-2.00x speedup while preserving generation quality, and our 4-step distilled model further delivers a 15.92x latency reduction with minimal visual quality drop.

Figures (10)

• Figure 1: Overview of the proposed efficient data-free post-training framework, LinVideo. (a) This framework first applies selective transfer (Sec. \ref{['sec:select_transfer']}), which assigns each layer a learnable score $r$ and progressively, automatically replaces quadratic attention with linear attention while minimizing the resulting performance drop. This process also combines with $\mathcal{L}_\text{con}$ (i.e., Eq. (\ref{['eq:constraint']})) and $\mathcal{L}_\text{reg}$ (i.e., Eq. (\ref{['eq:reg']})) to ensure a given target number of layers replaced by linear attention and mitigate the fluctuation (around $0.5$) of $r$ to improve training, respectively. (b) Moreover, LinVideo integrates an anytime distribution matching objective (Sec. \ref{['sec:tdm']}), which aims to match the sample distributions between $\hat{{\bm{u}}}_\theta$ and ${\bm{u}}_\theta$ across any timestep in the sampling trajectory. This significantly recovers performance and enables high efficiency compared with previous objectives in our scenarios.
• Figure 2: Performance on $4$ VBench huang2024vbench dimensions for partial linearized ($10$ adjacent layers for each dot) Wan$1.3$B wanteam2025wanopenadvancedlargescale after $2K$-step fine-tuning. The index range of the layers replaced with linear attention is indicated in the tick label of the x-axis. "*" denotes models further fine-tuned for $3K$ additional steps.
• Figure 3: Values of $r$ across layers and training steps. "w/ $\mathcal{L}_\text{reg}$" denotes we employ Eq. (\ref{['eq:reg']}) for training, otherwise only Eq. (\ref{['eq:constraint']}) is applied to guide the training of $r$.
• Figure 4: Performance comparison with baselines on VBench-2.0 zheng2025vbench20advancingvideogeneration. For Wan 1.3B, the total scores are $56.74$ (FA2), $55.81$ (SVG2), $56.74$ (LinVideo), and $55.51$ (LinVideo+DMD2); for Wan 14B, the total scores are $59.85$ (FA2), $58.74$ (SVG2), $59.62$ (LinVideo), and $58.22$ (LinVideo+DMD2). FA2 denotes FlashAttention2 dao2023flashattention2fasterattentionbetter.
• Figure 5: Visual results at $480p$ across Wan 1.3B wanteam2025wanopenadvancedlargescale (Upper), 1.3B LinVideo (Middle), and 1.3B LinVideo+ 4-step DMD2 yin2024improved (Lower). Prompt: "A wide pink flower field under a stormy twilight sky with a faint magical glow. Buds burst into luminous blossoms that spread in waves across the meadow. Above, massive black storm clouds roll hard and fast, with layered billows, shelf-cloud structure, and clear turbulence; inner lightning pulses for drama, no ground strikes. A few bioluminescent motes drift between flowers; faint aurora-like ribbons sit behind the storm."