Visual Attention Network

TL;DR

This work introduces Large Kernel Attention (LKA), a linear-attention mechanism that preserves 2D structure, reduces computational cost, and enables channel adaptability for vision. Built atop LKA, the Visual Attention Network (VAN) is a simple four-stage backbone that outperforms comparable CNNs and vision transformers on ImageNet-1K, COCO, ADE20K, and related tasks, while maintaining favorable efficiency. The authors demonstrate through extensive ablations and cross-task experiments that LKA’s decomposition (depthwise local conv, depthwise dilation Conv, and 1×1 channel conv) provides a robust balance of locality, long-range dependence, and adaptability, supported by Grad-CAM visualizations. The work suggests a shift in vision backbones toward hybrid attention designs that leverage both convolutional structure and self-attention benefits, and provides code for broader adoption.

Abstract

While originally designed for natural language processing tasks, the self-attention mechanism has recently taken various computer vision areas by storm. However, the 2D nature of images brings three challenges for applying self-attention in computer vision. (1) Treating images as 1D sequences neglects their 2D structures. (2) The quadratic complexity is too expensive for high-resolution images. (3) It only captures spatial adaptability but ignores channel adaptability. In this paper, we propose a novel linear attention named large kernel attention (LKA) to enable self-adaptive and long-range correlations in self-attention while avoiding its shortcomings. Furthermore, we present a neural network based on LKA, namely Visual Attention Network (VAN). While extremely simple, VAN surpasses similar size vision transformers(ViTs) and convolutional neural networks(CNNs) in various tasks, including image classification, object detection, semantic segmentation, panoptic segmentation, pose estimation, etc. For example, VAN-B6 achieves 87.8% accuracy on ImageNet benchmark and set new state-of-the-art performance (58.2 PQ) for panoptic segmentation. Besides, VAN-B2 surpasses Swin-T 4% mIoU (50.1 vs. 46.1) for semantic segmentation on ADE20K benchmark, 2.6% AP (48.8 vs. 46.2) for object detection on COCO dataset. It provides a novel method and a simple yet strong baseline for the community. Code is available at https://github.com/Visual-Attention-Network.

Figures (7)

• Figure 1: Results of different models on ImageNet-1K validation set. Comparing the performance of recent models DeiT touvron2021training, PVT wang2021pyramid, Swin Transformer liu2021swin, ConvNeXt liu2022convnet, Focal Transformer yang2021focal and our VAN. Above: Accuracy-Parameters trade-off diagram. Under: Accuracy-FLOPs trade-off diagram.
• Figure 2: Decomposition diagram of large-kernel convolution. A standard convolution can be decomposed into three parts: a depth-wise convolution (DW-Conv), a depth-wise dilation convolution (DW-D-Conv), and a pointwise convolution (1$\times$1 Conv). The colored grids represent the location of convolution kernel and the yellow grid means the center point. The diagram shows that a 13$\times$13 convolution is decomposed into a 5$\times$5 depth-wise convolution, a 5$\times$5 depth-wise dilation convolution with dilation rate 3, and a pointwise convolution. Note: zero paddings are omitted in the above figure.
• Figure 3: The structure of different modules: (a) the proposed Large Kernel Attention (LKA); (b) non-attention module; (c) replace multiplication in LKA with addition ; (d) self-attention. It is worth noting that (d) is designed for 1D sequences.
• Figure 4: A stage of VAN. d means depth wise convolution. k $\times$ k denotes k $\times$ k convolution.
• Figure 5: Accuracy-Throughput Diagram. It claerly shows that VAN achieves a better trade-off than swin transformer liu2021swin.