CSA-Net: Channel-wise Spatially Autocorrelated Attention Networks

TL;DR

CSA-Net introduces a novel channel-wise spatially autocorrelated (CSA) attention module that leverages local Moran's I-like spatial statistics to capture inter-channel spatial relationships with minimal overhead. The CSA descriptor is refined by a compact MLP to produce a channel attention map, which modulates feature maps as F = p ⊗ F and is integrated at multiple resolution scales. Empirical results on ImageNet and MS-COCO demonstrate competitive or superior performance to existing attention methods with lower computational cost, across image classification, object detection, and instance segmentation. This work demonstrates the practicality of applying geographical spatial analysis concepts to deep learning, enabling more discriminative and efficient channel representations.

Abstract

In recent years, convolutional neural networks (CNNs) with channel-wise feature refining mechanisms have brought noticeable benefits to modelling channel dependencies. However, current attention paradigms fail to infer an optimal channel descriptor capable of simultaneously exploiting statistical and spatial relationships among feature maps. In this paper, to overcome this shortcoming, we present a novel channel-wise spatially autocorrelated (CSA) attention mechanism. Inspired by geographical analysis, the proposed CSA exploits the spatial relationships between channels of feature maps to produce an effective channel descriptor. To the best of our knowledge, this is the f irst time that the concept of geographical spatial analysis is utilized in deep CNNs. The proposed CSA imposes negligible learning parameters and light computational overhead to the deep model, making it a powerful yet efficient attention module of choice. We validate the effectiveness of the proposed CSA networks (CSA-Nets) through extensive experiments and analysis on ImageNet, and MS COCO benchmark datasets for image classification, object detection, and instance segmentation. The experimental results demonstrate that CSA-Nets are able to consistently achieve competitive performance and superior generalization than several state-of-the-art attention-based CNNs over different benchmark tasks and datasets.

Figures (6)

• Figure 1: Conceptual comparison of the generic channel descriptors used by most of existing attention mechanisms senetGENetCBAMGALAdianet and the proposed one. Any image passing a convolutional layer is represented by a set of feature maps (Assume $f_{i}$,$f_{j}$, and $f_{k} \in \mathbb{R}^3$). Due to the data dependency of statistical pooling operators, feature maps show similar statistical properties while their spatial relationships are distinctive.
• Figure 2: The proposed CSA framework for modelling channel-wise attention maps. As illustrated, spatial autocorrelation of feature maps is utilized to refine global contextual information. Boxes with different colours demonstrate different values of the computed attention map upon the channel axis. $\mathbf{q}$ refers to the computed spatially autocorrelated channel descriptors. $\mathnormal{D}$ and $\mathnormal{U}$ indicate fully connected layers of the multi-layer perceptron (MLP) for channel reduction and up-sampling, respectively.
• Figure 3: Visualising the relationship between averaged channel-wise global context ($\mathbf{z}$), channel-wise spatial autocorrelation ($\mathbf{q}$), and corresponding channel attention map ($\mathbf{p}$) produced by the proposed CSA and SE SENet_j attention mechanisms in the last convolutional block at each resolution scale of ResNet-50 on the ImageNet validation set. Please note: the attention values for SE SENet_j are based on their global average pooling, while the proposed CSA is based on channels' spatial autocorrelation. For better presentation, the graphs are smoothed by the exponential moving average with a factor of $0.3$.
• Figure 4: Visual comparisons of Grad-CAM grad-cam analysis generated by the last convolutional outputs layer in ResNet-50, SE-ResNet-50 SENet_j, and CSA-ResNet-50 (Ours) on ImageNet. The target class label is shown on the top of each input image, and $p$ indicates the probability score of each model for the target class.
• Figure 5: Visualising the relationship between averaged channel-wise global context ($\mathbf{z}$), channel-wise spatial autocorrelation ($\mathbf{q}$), and corresponding channel attention map ($\mathbf{p}$) produced by the proposed CSA and SE SENet_j attention mechanisms in the last convolutional block at each resolution scale of ResNet-50 on the ImageNet validation set. Please note: the attention values for SE SENet_j are based on their global average pooling, while the proposed CSA is based on channels' spatial autocorrelation. For better presentation, the graphs are smoothed by the exponential moving average with a factor of $0.3$.