Interlaced Sparse Self-Attention for Semantic Segmentation

TL;DR

Interlaced Sparse Self-Attention for Semantic Segmentation introduces a method to factorize the dense affinity matrix into the product of two sparse affinity matrices, enabling long-range and short-range attention with reduced computation and memory. The approach provides two cascaded attention modules (long-range then short-range) so every position can integrate information from all others, and is validated across Cityscapes, ADE20K, LIP, and Mask-RCNN/COCO with consistent improvements over baselines and non-local methods. Comprehensive ablations elucidate the effects of partition size, attention order, and deeper stacking on performance. The work offers a practical, scalable alternative to dense self-attention for high-resolution semantic segmentation and related tasks.

Abstract

In this paper, we present a so-called interlaced sparse self-attention approach to improve the efficiency of the \emph{self-attention} mechanism for semantic segmentation. The main idea is that we factorize the dense affinity matrix as the product of two sparse affinity matrices. There are two successive attention modules each estimating a sparse affinity matrix. The first attention module is used to estimate the affinities within a subset of positions that have long spatial interval distances and the second attention module is used to estimate the affinities within a subset of positions that have short spatial interval distances. These two attention modules are designed so that each position is able to receive the information from all the other positions. In contrast to the original self-attention module, our approach decreases the computation and memory complexity substantially especially when processing high-resolution feature maps. We empirically verify the effectiveness of our approach on six challenging semantic segmentation benchmarks.

Figures (1)

• Figure 1: Visualization of predictions of FCN with and without our interlaced attention. The first and last two rows each present $4$ examples from the validation set of Cityscapes and ADE$20$K, respectively. Best viewed in color.