Cross-Layer Attentive Feature Upsampling for Low-latency Semantic Segmentation

TL;DR

This work tackles the need for high-resolution, semantically rich feature maps under real-time constraints in semantic segmentation. It introduces Guided Attentive Interpolation (GAI), an attention-based upsampling operator that links high-resolution spatial details with low-resolution semantic context through pixel-level relations, using Criss-Cross Attention to maintain efficiency. Integrated into the GAIN network, GAI produces high-resolution, spatially aligned, semantically enriched features that improve dense predictions while preserving low latency. Across Cityscapes, CamVid, ADE20K, and PASCAL Context, GAIN achieves state-of-the-art or competitive accuracy with significantly favorable speed, underscoring the practical impact for real-time scene understanding. The approach offers a general, plug-in enhancement for CNN-based segmentation that can be extended to other dense-prediction tasks.

Abstract

Semantic segmentation is a fundamental problem in computer vision and it requires high-resolution feature maps for dense prediction. Current coordinate-guided low-resolution feature interpolation methods, e.g., bilinear interpolation, produce coarse high-resolution features which suffer from feature misalignment and insufficient context information. Moreover, enriching semantics to high-resolution features requires a high computation burden, so that it is challenging to meet the requirement of lowlatency inference. We propose a novel Guided Attentive Interpolation (GAI) method to adaptively interpolate fine-grained high-resolution features with semantic features to tackle these issues. Guided Attentive Interpolation determines both spatial and semantic relations of pixels from features of different resolutions and then leverages these relations to interpolate high-resolution features with rich semantics. GAI can be integrated with any deep convolutional network for efficient semantic segmentation. In experiments, the GAI-based semantic segmentation networks, i.e., GAIN, can achieve78.8 mIoU with 22.3 FPS on Cityscapes and 80.6 mIoU with 64.5 on CamVid using an NVIDIA 1080Ti GPU, which are the new state-of-the-art results of low-latency semantic segmentation. Code and models are available at: https://github.com/hustvl/simpleseg.

Figures (8)

• Figure 1: Guided Attentive Interpolation. GAI will build the pixel-level pairwise relations between query points and key points from high-resolution features and low-resolution features respectively, and leverage the relations to interpolate high-resolution semantic features.
• Figure 2: Guided Attentive Interpolation Module. The low-resolution feature maps will be interpolated to the same size as the high-resolution feature maps. The concatenation of high-resolution and low-resolution feature maps is defined as the query. All $1\times1$ convolutions are used to reduce the dimension for less computation budget.
• Figure 3: The network architecture of our proposed GAIN (GAI-based Network) with two Guided Attentive Interpolation modules to interpolate features from {C4, C5} of ResNet (or DF-2) to $\times\frac{1}{8}$ resolution for fusion as the fine-grained semantic features. GAP denotes the global average pooling. All convolutions are $1\times1$ for less computation budget.
• Figure 4: Speed-accuracy trade-off. Our methods are presented in stars and triangles for different backbones. Other methods are presented in blue circles. Our proposed GAIN achieves a superior trade-off between speed and segmentation accuracy.
• Figure 5: Visualizations of feature maps before/after Guided Attentive Interpolation modules. {C3, C4, C5} are the output features from different stages of the backbone (C3 contains higher-resolution feature maps). For feature visualization, we perform an element-wise sum along the channel axis for each $C$-channel features to obtain a single-channel feature map. The lighter area has a higher response.