Threshold Attention Network for Semantic Segmentation of Remote Sensing Images

TL;DR

This paper tackles the high computational cost and redundancy of self-attention in semantic segmentation of high-resolution remote sensing images. It introduces the Threshold Attention Mechanism (TAM), which operates on region-level thresholds to achieve linear-complexity attention, improving efficiency and feature quality. Building on TAM, the authors propose TANet, featuring AFEM for shallow-feature enhancement and TAPP for multi-scale deep-feature aggregation, fused to yield accurate and detailed segmentation maps. Experiments on ISPRS Vaihingen and Potsdam demonstrate state-of-the-art performance with substantially fewer parameters and lower computational burden than traditional self-attention-based methods, indicating strong practical potential for large-scale remote sensing analysis.

Abstract

Semantic segmentation of remote sensing images is essential for various applications, including vegetation monitoring, disaster management, and urban planning. Previous studies have demonstrated that the self-attention mechanism (SA) is an effective approach for designing segmentation networks that can capture long-range pixel dependencies. SA enables the network to model the global dependencies between the input features, resulting in improved segmentation outcomes. However, the high density of attentional feature maps used in this mechanism causes exponential increases in computational complexity. Additionally, it introduces redundant information that negatively impacts the feature representation. Inspired by traditional threshold segmentation algorithms, we propose a novel threshold attention mechanism (TAM). This mechanism significantly reduces computational effort while also better modeling the correlation between different regions of the feature map. Based on TAM, we present a threshold attention network (TANet) for semantic segmentation. TANet consists of an attentional feature enhancement module (AFEM) for global feature enhancement of shallow features and a threshold attention pyramid pooling module (TAPP) for acquiring feature information at different scales for deep features. We have conducted extensive experiments on the ISPRS Vaihingen and Potsdam datasets. The results demonstrate the validity and superiority of our proposed TANet compared to the most state-of-the-art models.

Figures (8)

• Figure 1: Traditional threshold segmentation method. (a) is the original remote sensing image. (b) is the image histogram of (a) in the red channel. (c) is the segmentation result map obtained by the traditional threshold segmentation method. (d) is the label map of (a).
• Figure 2: TANet utilizes the ResNet101 backbone network to extract features. Additionally, it employs the AFEM module to enhance the feature information obtained from the shallow network, and the TAPP module to capture rich global semantic information from the deep features. Subsequently, the two complementary feature maps are integrated to acquire a consolidated feature map. Ultimately, bilinear interpolation is leveraged to generate the ultimate predicted output map.
• Figure 3: TAM consists of three parts: (a) for thresholding the input features, (b) for calculating the attention weight matrix, and (c) for recovering location information for features. TAM is an attention mechanism that exhibits linear computational complexity and effectively models the correlation between similar regions in the feature map.
• Figure 4: The structure of the AFEM is composed of three branches: the first one acquires channel attention, the second one enhances threshold attentional features, and the third one provides residual connectivity.
• Figure 5: Structure of the TAPP, where CBR is the convolution layer + BN layer + ReLU layer, Cos is the calculation of cosine similarity, and GAP is the calculation of global average pooling.