Dynamic Differential Linear Attention: Enhancing Linear Diffusion Transformer for High-Quality Image Generation

TL;DR

This work addresses the quadratic bottleneck of self-attention in diffusion transformers by introducing Dynamic Differential Linear Attention (DyDiLA), a linear-attention formulation composed of a dynamic projection module, a dynamic measure kernel, and a token differential operator. Built into a refined diffusion model, DyDi-LiT, it preserves linear complexity while sharpening attention, improving token disentanglement, similarity measurement, and query–key retrieval. Extensive experiments on Sub-IN and ImageNet-1K across 256×256 and 512×512 resolutions show DyDi-LiT achieving state-of-the-art results among efficient diffusion models and closing the gap to quadratic-attention baselines. The results demonstrate significant practical impact for scalable, high-fidelity image generation with reduced compute, enabling higher-resolution synthesis without prohibitive costs.

Abstract

Diffusion transformers (DiTs) have emerged as a powerful architecture for high-fidelity image generation, yet the quadratic cost of self-attention poses a major scalability bottleneck. To address this, linear attention mechanisms have been adopted to reduce computational cost; unfortunately, the resulting linear diffusion transformers (LiTs) models often come at the expense of generative performance, frequently producing over-smoothed attention weights that limit expressiveness. In this work, we introduce Dynamic Differential Linear Attention (DyDiLA), a novel linear attention formulation that enhances the effectiveness of LiTs by mitigating the oversmoothing issue and improving generation quality. Specifically, the novelty of DyDiLA lies in three key designs: (i) dynamic projection module, which facilitates the decoupling of token representations by learning with dynamically assigned knowledge; (ii) dynamic measure kernel, which provides a better similarity measurement to capture fine-grained semantic distinctions between tokens by dynamically assigning kernel functions for token processing; and (iii) token differential operator, which enables more robust query-to-key retrieval by calculating the differences between the tokens and their corresponding information redundancy produced by dynamic measure kernel. To capitalize on DyDiLA, we introduce a refined LiT, termed DyDi-LiT, that systematically incorporates our advancements. Extensive experiments show that DyDi-LiT consistently outperforms current state-of-the-art (SOTA) models across multiple metrics, underscoring its strong practical potential.

Figures (11)

• Figure 1: Inference cost and performance comparisons among DiT dit using Softmax attention, Sana xie2024sana using linear attention, and our DyDi-LiT using DyDiLA. DyDiLA achieves SOTA performance with negligible additional computational overhead.
• Figure 2: Overview of DyDi-LiT. (a) DyDi-LiT comprises $L$ blocks that receive noise tokens encoded by VAE, and AdaLN injects timestep and class information into every block. (b) DyDiLA comprises three components---dynamic projection module, dynamic measure kernel, and token differential operator---responsible respectively for disentangling token representations, providing more accurate token similarity measurement, and strengthening query–key retrieval robustness.
• Figure 3: Generation results of models on Sub-IN benchmark at 256$\times$256 resolution. CFG scale is 4.0. As the model size increases, the generation quality consistently improves. Overall, DyDi-LiT produces the highest-quality images. Best viewed zoomed-in.
• Figure 4: Generation results of large version models on Sub-IN benchmark at 512$\times$512 resolution. CFG scale is 4.0. Overall, DyDi-LiT produces the highest-quality images. Best viewed zoomed-in.
• Figure 5: Routing visualization of dynamic projection module. For images from various categories, the most frequently accessed projector in each block is shown, revealing category-specific routing and disentangled token representations.