Can We Achieve Efficient Diffusion without Self-Attention? Distilling Self-Attention into Convolutions

TL;DR

Diffusion models incur heavy quadratic cost from self-attention, but attention in practice is largely local. The authors decompose this behavior into a high-frequency, distance-dependent local signal and a low-frequency global bias, then replace self-attention with ΔConvBlocks built from Pyramid Convolutions plus an attention-bias path, trained via feature- and output-level distillation with MinSNR weighting. Across U-Net and DiT backbones, ΔConvFusion achieves comparable fidelity while dramatically reducing computation, including up to $6929\times$ FLOPs reduction at 16{,}000-pixel resolution and strong cross-resolution performance. This approach enables efficient, scalable diffusion without sacrificing image quality or semantic alignment, and generalizes across architectures and resolutions, offering practical impact for deployable diffusion systems.

Abstract

Contemporary diffusion models built upon U-Net or Diffusion Transformer (DiT) architectures have revolutionized image generation through transformer-based attention mechanisms. The prevailing paradigm has commonly employed self-attention with quadratic computational complexity to handle global spatial relationships in complex images, thereby synthesizing high-fidelity images with coherent visual semantics.Contrary to conventional wisdom, our systematic layer-wise analysis reveals an interesting discrepancy: self-attention in pre-trained diffusion models predominantly exhibits localized attention patterns, closely resembling convolutional inductive biases. This suggests that global interactions in self-attention may be less critical than commonly assumed.Driven by this, we propose \(Δ\)ConvFusion to replace conventional self-attention modules with Pyramid Convolution Blocks (\(Δ\)ConvBlocks).By distilling attention patterns into localized convolutional operations while keeping other components frozen, \(Δ\)ConvFusion achieves performance comparable to transformer-based counterparts while reducing computational cost by 6929$\times$ and surpassing LinFusion by 5.42$\times$ in efficiency--all without compromising generative fidelity.

Figures (9)

• Figure 1: Computational cost comparison between self-attention and our $\Delta$ConvBlock in SD1.5 LDM. Our method significantly reduces FLOPs, achieving up to $\mathbf{6929\times}$ lower computational cost at 16K resolution while maintaining performance.
• Figure 2: Attention visualization across different layers for FLUX, PixArt, and SD1.5. Each column shows attention maps at different layers for a given query location in the input image. The red circles highlight artifacts in PixArt's attention maps (abnormally focus on the last pixel), indicating irregular attention distributions. Across all models, most pixels exhibit highly localized attention, suggesting that self-attention predominantly operates within local neighborhoods rather than modeling global interactions.
• Figure 3: Overall diffusion architectures of U-Net and our proposed $\Delta$ConvFusion. We replace self-attention modules in Transformer U-Net with $\Delta$ConvBlocks, yielding $\Delta$ConvFusion (U-Net).
• Figure 4: $\mathtt{ASM}$ and its gradient across kernels with varying sizes. From (a), both PixArt and SD1.5 exhibit a consistent quadratic relationship between kernel size $K$ and $\mathtt{ASM}$. This indicates that attention primarily captures broad low-frequency information, as the accumulated attention follows the typical scaling behavior of spatially smooth signals. From (b), the $\mathtt{ASM}$ of high-pass filtered attention maps decays quadratically with increasing distance from the query pixel, reinforcing its strong locality. For SD1.5 (U-Net architecture), $\mathtt{ASM}$ is computed at $64 \times 64$ and $32 \times 32$ scales, while PixArt uses a fixed $64 \times 64$ attention map.
• Figure 5: Effective Receptive Field (ERF) of self-attention across layers in PixArt-Sigma and SD1.5. PixArt shows larger ERFs in certain layers due to artifacts (Fig. 2) but remains below $15 \times 15$ in most cases, while SD1.5 stays under $20 \times 20$. These insights guide $\Delta$ConvBlock kernel size selection.