$A^2$-Nets: Double Attention Networks

TL;DR

Long-range dependencies are challenging for standard CNNs. The paper introduces A^2-Net, a lightweight double-attention block that first gathers global features via second-order attention pooling and then distributes them adaptively to each location, enabling CNN layers to access global context efficiently. Empirical results on ImageNet-1k and video benchmarks (Kinetics, UCF-101) show that A^2-Net improves accuracy with substantially fewer parameters and FLOPs than strong baselines and non-local methods. The work provides a generic gathering-distribution formulation and demonstrates its effectiveness across image and video recognition, with potential for integration into compact, mobile-friendly architectures.

Abstract

Learning to capture long-range relations is fundamental to image/video recognition. Existing CNN models generally rely on increasing depth to model such relations which is highly inefficient. In this work, we propose the "double attention block", a novel component that aggregates and propagates informative global features from the entire spatio-temporal space of input images/videos, enabling subsequent convolution layers to access features from the entire space efficiently. The component is designed with a double attention mechanism in two steps, where the first step gathers features from the entire space into a compact set through second-order attention pooling and the second step adaptively selects and distributes features to each location via another attention. The proposed double attention block is easy to adopt and can be plugged into existing deep neural networks conveniently. We conduct extensive ablation studies and experiments on both image and video recognition tasks for evaluating its performance. On the image recognition task, a ResNet-50 equipped with our double attention blocks outperforms a much larger ResNet-152 architecture on ImageNet-1k dataset with over 40% less the number of parameters and less FLOPs. On the action recognition task, our proposed model achieves the state-of-the-art results on the Kinetics and UCF-101 datasets with significantly higher efficiency than recent works.

Figures (2)

• Figure 1: Illustration of the double-attention mechanism. (a) An example on a single frame input for explaining the idea of our double attention method, where the set of global featues is computed only once and then shared by all locations. Meanwhile, each location $i$ will generate its own attention vector based on the need of its local feature $\mathbf{v}_i$ to select a desired subset of global features that is helpful to complement current location and form the feature $\mathbf{z}_i$. (b) The double attention operation on a three dimensional input array $A$. The first attention step is shown on the top and produces a set of global features. At location $i$, the second attention step generates the new local feature $\mathbf{z}_i$, as shown at the bottom.
• Figure 2: The computational graph of the proposed double attention block. All convolution kernel size is $1\times1\times1$. We insert this double attention block to existing convolutional neural network, e.g. residual networks he2016deep, to form the $A^2$-Net.