Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

TL;DR

The paper tackles semantic image segmentation by marrying multi-scale contextual encoding with a boundary-refining decoder in an encoder-decoder framework. It introduces DeepLabv3+, extends the encoder with atrous convolutions and an ASPP-based context module, and adds a simple yet effective decoder; it also adopts a Modified Aligned Xception backbone and applies atrous separable convolutions for efficiency. Through extensive experiments on PASCAL VOC 2012 and Cityscapes, it demonstrates significant gains in accuracy and speed, achieving state-of-the-art results without post-processing, and provides public TensorFlow code. The work highlights the importance of boundary restoration in segmentation and shows how architectural choices and efficient backbones can yield practical, high-performance models for real-world datasets.

Abstract

Spatial pyramid pooling module or encode-decoder structure are used in deep neural networks for semantic segmentation task. The former networks are able to encode multi-scale contextual information by probing the incoming features with filters or pooling operations at multiple rates and multiple effective fields-of-view, while the latter networks can capture sharper object boundaries by gradually recovering the spatial information. In this work, we propose to combine the advantages from both methods. Specifically, our proposed model, DeepLabv3+, extends DeepLabv3 by adding a simple yet effective decoder module to refine the segmentation results especially along object boundaries. We further explore the Xception model and apply the depthwise separable convolution to both Atrous Spatial Pyramid Pooling and decoder modules, resulting in a faster and stronger encoder-decoder network. We demonstrate the effectiveness of the proposed model on PASCAL VOC 2012 and Cityscapes datasets, achieving the test set performance of 89.0\% and 82.1\% without any post-processing. Our paper is accompanied with a publicly available reference implementation of the proposed models in Tensorflow at \url{https://github.com/tensorflow/models/tree/master/research/deeplab}.

Figures (6)

• Figure 1: We improve DeepLabv3, which employs the spatial pyramid pooling module (a), with the encoder-decoder structure (b). The proposed model, DeepLabv3+, contains rich semantic information from the encoder module, while the detailed object boundaries are recovered by the simple yet effective decoder module. The encoder module allows us to extract features at an arbitrary resolution by applying atrous convolution.
• Figure 2: Our proposed DeepLabv3+ extends DeepLabv3 by employing a encoder-decoder structure. The encoder module encodes multi-scale contextual information by applying atrous convolution at multiple scales, while the simple yet effective decoder module refines the segmentation results along object boundaries.
• Figure 3: $3\times3$ Depthwise separable convolution decomposes a standard convolution into (a) a depthwise convolution (applying a single filter for each input channel) and (b) a pointwise convolution (combining the outputs from depthwise convolution across channels). In this work, we explore atrous separable convolution where atrous convolution is adopted in the depthwise convolution, as shown in (c) with $rate=2$.
• Figure 4: We modify the Xception as follows: (1) more layers (same as MSRA's modification except the changes in Entry flow), (2) all the max pooling operations are replaced by depthwise separable convolutions with striding, and (3) extra batch normalization and ReLU are added after each $3\times3$ depthwise convolution, similar to MobileNet.
• Figure 5: (a) mIOU as a function of trimap band width around the object boundaries when employing $\emph{train\ output\ stride} = \emph{eval\ output\ stride} = 16$. BU: Bilinear upsampling. (b) Qualitative effect of employing the proposed decoder module compared with the naive bilinear upsampling (denoted as BU). In the examples, we adopt Xception as feature extractor and $\emph{train\ output\ stride} = \emph{eval\ output\ stride} = 16$.