SCSA: Exploring the Synergistic Effects Between Spatial and Channel Attention

TL;DR

The results demonstrate that the proposed SCSA not only surpasses the current state-of-the-art attention but also exhibits enhanced generalization capabilities across various task scenarios.

Abstract

Channel and spatial attentions have respectively brought significant improvements in extracting feature dependencies and spatial structure relations for various downstream vision tasks. While their combination is more beneficial for leveraging their individual strengths, the synergy between channel and spatial attentions has not been fully explored, lacking in fully harness the synergistic potential of multi-semantic information for feature guidance and mitigation of semantic disparities. Our study attempts to reveal the synergistic relationship between spatial and channel attention at multiple semantic levels, proposing a novel Spatial and Channel Synergistic Attention module (SCSA). Our SCSA consists of two parts: the Shareable Multi-Semantic Spatial Attention (SMSA) and the Progressive Channel-wise Self-Attention (PCSA). SMSA integrates multi-semantic information and utilizes a progressive compression strategy to inject discriminative spatial priors into PCSA's channel self-attention, effectively guiding channel recalibration. Additionally, the robust feature interactions based on the self-attention mechanism in PCSA further mitigate the disparities in multi-semantic information among different sub-features within SMSA. We conduct extensive experiments on seven benchmark datasets, including classification on ImageNet-1K, object detection on MSCOCO 2017, segmentation on ADE20K, and four other complex scene detection datasets. Our results demonstrate that our proposed SCSA not only surpasses the current state-of-the-art attention but also exhibits enhanced generalization capabilities across various task scenarios. The code and models are available at: https://github.com/HZAI-ZJNU/SCSA.

Figures (8)

• Figure 1: Visualization of several feature maps. Different box or circle colors highlight inherent spatial semantic disparities across specific parts in various feature maps.
• Figure 2: An illustration of our proposed SCSA, which uses multi-semantic spatial information to guide the learning of channel-wise self-attention. $B$ denotes the batch size, $C$ signifies the number of channels, and $H$ and $W$ correspond to the height and width of the feature maps, respectively. The variable $n$ represents the number of groups into which sub-features are divided, and $1P$ denotes a single pixel.
• Figure 3: Main Module Structures with SCSA
• Figure 4: Comparative attention visualizations for 'layer 4.2' across multiple models, generated using samples randomly selected from different categories of the ImageNet-1K validation set, through Grad-CAM GradCAM.
• Figure 5: Comparison of effective receptive fields (ERFs). Our SCSA provides a larger effective receptive field compared to the baseline, and the effect becomes more pronounced as the layers deepen.