Attentive Convolution: Unifying the Expressivity of Self-Attention with Convolutional Efficiency

TL;DR

Self-attention incurs a quadratic compute cost $O(N^2)$ on input size $N$, while convolution is typically linear in $N$. The paper identifies adaptive routing and lateral inhibition as the two core principles behind SA's expressivity and shows how to transplant them into a convolutional operator via Context-to-Kernel Translation, learnable value projection, and Differential Kernel Modulation to form ATConv. AttNet, built from ATConv, achieves 84.4% ImageNet-1K Top-1 with 27M parameters and delivers diffusion-generation improvements (FID reductions and faster sampling) compared with SA baselines. This work provides a scalable, efficient backbone operator that bridges CNNs and ViTs across classification, detection, segmentation, robustness, and generation tasks.

Abstract

Self-attention (SA) has become the cornerstone of modern vision backbones for its powerful expressivity over traditional Convolutions (Conv). However, its quadratic complexity remains a critical bottleneck for practical applications. Given that Conv offers linear complexity and strong visual priors, continuing efforts have been made to promote the renaissance of Conv. However, a persistent performance chasm remains, highlighting that these modernizations have not yet captured the intrinsic expressivity that defines SA. In this paper, we re-examine the design of the CNNs, directed by a key question: what principles give SA its edge over Conv? As a result, we reveal two fundamental insights that challenge the long-standing design intuitions in prior research (e.g., Receptive field). The two findings are: (1) \textit{Adaptive routing}: SA dynamically regulates positional information flow according to semantic content, whereas Conv employs static kernels uniformly across all positions. (2) \textit{Lateral inhibition}: SA induces score competition among token weighting, effectively suppressing redundancy and sharpening representations, whereas Conv filters lack such inhibitory dynamics and exhibit considerable redundancy. Based on this, we propose \textit{Attentive Convolution} (ATConv), a principled reformulation of the convolutional operator that intrinsically injects these principles. Interestingly, with only $3\times3$ kernels, ATConv consistently outperforms various SA mechanisms in fundamental vision tasks. Building on ATConv, we introduce AttNet, a CNN family that can attain \textbf{84.4\%} ImageNet-1K Top-1 accuracy with only 27M parameters. In diffusion-based image generation, replacing all SA with the proposed $3\times 3$ ATConv in SiT-XL/2 reduces ImageNet FID by 0.15 in 400k steps with faster sampling. Code is available at: github.com/price112/Attentive-Convolution.

Figures (6)

• Figure 1: Visualization of adaptive routing and lateral inhibition on 3$\times$3 ATConv, 7$\times$7 DWConv liu2022convnet, 3$\times$3 Conv he2016deep, vanilla global attention dosovitskiy2020image, window liu2021swin and pooling wang2021pyramid attention. We analyze operator-intrinsic behavior by extracting influence maps $G(h,w)=\sum_c \lvert \partial y_{c^\ast,h^\ast,w^\ast}/\partial x_{c,h,w}\rvert$. Adaptive routing is visualized via distance-weighted centroids of high-influence regions, with $\mathrm{FAR}$ at radial threshold $r_0$ measuring long-range preference. Diverse arrows indicate different inputs produce different aggregation patterns, while uniform arrows reveal fixed routing. Lateral inhibition is quantified by the off-center suppression response to positive perturbations at anchors, where brighter regions indicate stronger surround suppression. ATConv with a compact 3$\times$3 kernel exhibits pronounced adaptive routing and lateral inhibition effects similar to the global self-attention. See our depository for detailed visualization code.
• Figure 2: (a) Overall architecture of AttNet, where we use ATConv as token-mixer (spatial-operator) and GLU as channel-mixer; (b) Architecture of Attentive Convolution (ATConv); (c) Architecture of kernel generator which builds the initial kernels for ATConv based on input contents.
• Figure 3: Impact of Differential Kernel Modulation (Diff. Modulation) on feature properties. We analyze the output of every ATConv in AttNet-T4 with and without using Diff. Modulation. (a) Center-Surround Contrast is defined as $\mathrm{CSC}(x) = \mathbb{E}[|x - G_\sigma(x)|]\cdot\mathbb{E}[|x|]^{-1}$, where $G_\sigma(\cdot)$ is Gaussian blurring. CSC quantifies the relative enhancement of local pattern contrast, higher means sharper representations. (b) Channel Effective Rank is defined as $\mathrm{CER}(x) = \exp(H(p))\cdot C^{-1}$, where $p_i = \lambda_i / \sum_j \lambda_j$ are normalized eigenvalues of the channel covariance and $H(p) = -\sum_i p_i \log p_i$. CER measures the intrinsic dimensionality of channel activations normalized by channel count, indicating diversity of representations, higher means better diversity and less flat response. Both metrics show consistent improvement across all layers with differential kernel modulation, demonstrating enhanced feature sharpness and diversity.
• Figure 4: (a): Latency vs. accuracy curve on PVT wang2021pyramid and Swin transformers liu2021swin, with their ATConv versions which directly replace the self-attention blocks with ATConv; (b) and (c): Latency and GPU memory consumption in terms of different input resolutions on different operators at the atomic level.
• Figure 5: Representative images generated by SiT-ATConv-XL/2, where all attention mechanisms are replaced with ATConv to form a pure convolutional generative architecture. The top two rows present natural images synthesized on the ImageNet-1K dataset using classifier-free guidance ($w=4.0$). The bottom row shows facial images unconditionally synthesized on the FFHQ dataset. We show results at 400K steps, more training steps will yield better quality.