DeeperLab: Single-Shot Image Parser

TL;DR

DeeperLab presents a single-shot, bottom-up approach for panoptic segmentation that jointly predicts semantic and instance labels in a fully convolutional framework. By sharing a decoder, employing depthwise separable convolutions, large kernels, space-to-depth transformations, and hard pixel mining, it achieves a favorable balance of accuracy and speed, demonstrated on Mapillary Vistas with competitive PQ and PC scores. The work also introduces Parsing Covering (PC), a region-based metric that complements PQ by emphasizing region size and real-world parsing quality. Collectively, these contributions enable faster, memory-efficient image parsing suitable for high-resolution scenes and real-time applications, with open-source potential for broader adoption.

Abstract

We present a single-shot, bottom-up approach for whole image parsing. Whole image parsing, also known as Panoptic Segmentation, generalizes the tasks of semantic segmentation for 'stuff' classes and instance segmentation for 'thing' classes, assigning both semantic and instance labels to every pixel in an image. Recent approaches to whole image parsing typically employ separate standalone modules for the constituent semantic and instance segmentation tasks and require multiple passes of inference. Instead, the proposed DeeperLab image parser performs whole image parsing with a significantly simpler, fully convolutional approach that jointly addresses the semantic and instance segmentation tasks in a single-shot manner, resulting in a streamlined system that better lends itself to fast processing. For quantitative evaluation, we use both the instance-based Panoptic Quality (PQ) metric and the proposed region-based Parsing Covering (PC) metric, which better captures the image parsing quality on 'stuff' classes and larger object instances. We report experimental results on the challenging Mapillary Vistas dataset, in which our single model achieves 31.95% (val) / 31.6% PQ (test) and 55.26% PC (val) with 3 frames per second (fps) on GPU or near real-time speed (22.6 fps on GPU) with reduced accuracy.

Figures (13)

• Figure 1: The proposed single-shot, bottom-up image parser, DeeperLab. The per-pixel semantic and instance predictions are generated using a single pass of a fully-convolutional network. These predictions are then fused into the final image parsing result by a fast algorithm.
• Figure 2: The proposed single-shot, bottom-up network architecture employs the encoder-decoder structure and produces per-pixel semantic and instance predictions. The number of channels of each feature map is specified in the figure.
• Figure 3: An example of the space-to-depth (S2D) and depth-to-space (D2S) operations. The S2D operation moves activations from the spatial dimension to the channel dimension and the D2S operation is the inverse.
• Figure 4: Four prediction maps generated by our instance-related heads: (a) keypoint heatmap, (b) long-range offset, (c) short-range offset, and (d) middle-range offset. The red stars denote the keypoints, the green disk denotes the target for keypoint prediction, and the blue lines/arrows denote the offsets from the current pixel to the target keypoint.
• Figure 5: A few image parsing results on the Mapillary Vistas validation set with proposed DeeperLab based on Xception-71. The first row is the predicted semantic segmentation and the second row is the predicted instance segmentation. Note that our model does not generate any VOID labels.