TransNeXt: Robust Foveal Visual Perception for Vision Transformers

TL;DR

TransNeXt tackles depth degradation in Vision Transformers by replacing deep stacking with Aggregated Attention that fuses foveal (sliding-window) and peripheral (pooled) perceptions, enhanced by learnable tokens and a Convolutional GLU channel mixer. The method yields a four-stage TransNeXt backbone that achieves state-of-the-art ImageNet results and robustness (including ImageNet-A) while delivering strong COCO detection and ADE20K segmentation performance, all with linear-like inference complexity under fixed pooling. Key contributions include Pixel-focused Attention, Aggregated Attention with learnable tokens (LKV/QLV), length-scaled cosine attention for multi-scale extrapolation, and ConvGLU for localized channel attention. Together, these components deliver a robust, biomimetic vision backbone with strong cross-task performance and practical efficiency, supported by a CUDA-accelerated implementation.

Abstract

Due to the depth degradation effect in residual connections, many efficient Vision Transformers models that rely on stacking layers for information exchange often fail to form sufficient information mixing, leading to unnatural visual perception. To address this issue, in this paper, we propose Aggregated Attention, a biomimetic design-based token mixer that simulates biological foveal vision and continuous eye movement while enabling each token on the feature map to have a global perception. Furthermore, we incorporate learnable tokens that interact with conventional queries and keys, which further diversifies the generation of affinity matrices beyond merely relying on the similarity between queries and keys. Our approach does not rely on stacking for information exchange, thus effectively avoiding depth degradation and achieving natural visual perception. Additionally, we propose Convolutional GLU, a channel mixer that bridges the gap between GLU and SE mechanism, which empowers each token to have channel attention based on its nearest neighbor image features, enhancing local modeling capability and model robustness. We combine aggregated attention and convolutional GLU to create a new visual backbone called TransNeXt. Extensive experiments demonstrate that our TransNeXt achieves state-of-the-art performance across multiple model sizes. At a resolution of $224^2$, TransNeXt-Tiny attains an ImageNet accuracy of 84.0%, surpassing ConvNeXt-B with 69% fewer parameters. Our TransNeXt-Base achieves an ImageNet accuracy of 86.2% and an ImageNet-A accuracy of 61.6% at a resolution of $384^2$, a COCO object detection mAP of 57.1, and an ADE20K semantic segmentation mIoU of 54.7.

Figures (9)

• Figure 1: A comprehensive comparison of performance on ImageNet-1K, robustness on ImageNet-A, COCO detection and instance segmentation performance based on Mask R-CNN $1\times$, ADE20K semantic segmentation performance based on UperNet.
• Figure 2: A comparison of prevalent visual information aggregation mechanisms, our proposed method, and biological visual systems (Left) and a visualization comparison of the Effective Receptive Field DBLP:conf/nips/LuoLUZ16 between our method and the prevalent backbone networks, using the output at stage 3 (Right). Each ERF image is generated by averaging over 5000 $224^2$-sized images from ImageNet-1K validation set.
• Figure 3: An illustration of the comparison between pixel-focused attention (left) and aggregated attention (right). Both have a feature size of 10$\times$10, a window size of 3$\times$3, and a pool size of 2$\times$2.
• Figure 4: Comparison of prevalent channel mixer designs and Convolutional GLU
• Figure 5: An illustration of TrasnNeXt architecture.