SALAD: Achieve High-Sparsity Attention via Efficient Linear Attention Tuning for Video Diffusion Transformer

TL;DR

SALAD tackles the efficiency bottleneck of long-sequence attention in video diffusion transformers by introducing a parallel linear attention branch that complements sparse attention. An input-dependent scalar gate dynamically balances the two branches, enabling ultra-sparse attention with high-quality video generation. Empirical results show SALAD reaches $90\%$ sparsity and a $1.72\times$ speedup while maintaining dense-attention-level quality, with highly efficient finetuning using only $2{,}000$ video samples and $1{,}600$ steps. This approach offers a practical path to scalable, fast video diffusion with minimal training overhead, suitable for deployment in real-world, large-sequence settings.

Abstract

Diffusion Transformers have recently demonstrated remarkable performance in video generation. However, the long input sequences result in high computational latency due to the quadratic complexity of full attention. Various sparse attention mechanisms have been proposed. Training-free sparse attention is constrained by limited sparsity and thus offers modest acceleration, whereas training-based methods can reach much higher sparsity but demand substantial data and computation for training. In this work, we propose SALAD, introducing a lightweight linear attention branch in parallel with the sparse attention. By incorporating an input-dependent gating mechanism to finely balance the two branches, our method attains 90% sparsity and 1.72x inference speedup, while maintaining generation quality comparable to the full attention baseline. Moreover, our finetuning process is highly efficient, requiring only 2,000 video samples and 1,600 training steps with a batch size of 8.

Figures (17)

• Figure 1: Comparison of Our Method and Other Sparse Attention Mechanisms. Score versus speedup, with point size representing density—smaller points indicate lower computational density. The Summation Score is computed as summation of VBench metrics (Subject Consistency, Background Consistency, Imaging Quality, and Text Consistency). This reflects overall quality–efficiency trade-off. Models compared include our approach, SVG2 yang2025sparse, PARO zhao2025paroattention, ST-SWA zhang2025ditfastattnv2, and ST-SWA + LoRA hu2022lora.
• Figure 2: Comparison of Full Attention Model, Sparse Attention Model and Sparse Attention Model with LoRA Tuning.
• Figure 3: Overview of SALAD Attention Module.
• Figure 4: The Rank of Sparse Attention and Linear Attention.
• Figure 5: Effect of Scaling the Linear Attention Branch. As $\lambda$ decreases, we observe improvements in background consistency, imaging quality, and text consistency. This suggests that the contribution of the linear attention branch to the final output must be carefully constrained. Relying solely on a static projection layer to regulate its influence appears insufficient, highlighting the need for more dynamic control over the branch fusion ratio.