SemSegDepth: A Combined Model for Semantic Segmentation and Depth Completion

TL;DR

This paper proposes a new end-to-end model for performing semantic segmentation and depth completion jointly and relies on RGB and sparse depth as inputs to this model and produces a dense depth map and the corresponding semantic segmentation image.

Abstract

Holistic scene understanding is pivotal for the performance of autonomous machines. In this paper we propose a new end-to-end model for performing semantic segmentation and depth completion jointly. The vast majority of recent approaches have developed semantic segmentation and depth completion as independent tasks. Our approach relies on RGB and sparse depth as inputs to our model and produces a dense depth map and the corresponding semantic segmentation image. It consists of a feature extractor, a depth completion branch, a semantic segmentation branch and a joint branch which further processes semantic and depth information altogether. The experiments done on Virtual KITTI 2 dataset, demonstrate and provide further evidence, that combining both tasks, semantic segmentation and depth completion, in a multi-task network can effectively improve the performance of each task. Code is available at https://github.com/juanb09111/semantic depth.

Figures (9)

• Figure 1: Overview of our proposed SemSegDepth architecture. Our model produces a dense depth map and semantics prediction given an RGB image and sparse depth as input.
• Figure 2: Diagram of the SemSegDepth architecture. The convolutional layers shown in this diagram follow the notation Conv($k$,$s$,$c$) where $k$ refers to a $k \times k$ convolutional kernel, $s$ is the stride, and $c$ is the number of output feature channels.
• Figure 3: Semantic Segmentation results on Virtual KITTI 2. Column a shows the RGB image input, column b shows the semantic segmentation results using the baseline model SemSegNet_b and column c shows the the semantic segmentation results using our model SemSegDepth.
• Figure 4: Depth completion results on Virtual KITTI 2. Column a shows the RGB image input, column b shows the depth completion results using the baseline model DepthNet_b and column c shows the the depth completion results using our model SemSegDepth.
• Figure 5: SemNet_depth_gt architecture.