S$^3$M-Net: Joint Learning of Semantic Segmentation and Stereo Matching for Autonomous Driving

TL;DR

S^3M-Net presents a unified framework for jointly learning semantic segmentation and stereo matching in autonomous driving by sharing RGB features, aligning them with semantic space via a feature fusion adaptation module, and supervising with a semantic consistency-guided loss. The approach combines a joint encoder, multi-level GRU-based disparity refinement, and a densely connected decoder to produce accurate segmentation while refining depth, with SCG loss enforcing structural coherence across tasks. Experiments on vKITTI2 and KITTI demonstrate state-of-the-art performance for both tasks under limited data, highlighting the benefits of cross-task regularization and feature fusion. The work offers a practical pathway toward resource-efficient, end-to-end perception systems, though it notes data requirements and runtime speed as areas for further optimization.

Abstract

Semantic segmentation and stereo matching are two essential components of 3D environmental perception systems for autonomous driving. Nevertheless, conventional approaches often address these two problems independently, employing separate models for each task. This approach poses practical limitations in real-world scenarios, particularly when computational resources are scarce or real-time performance is imperative. Hence, in this article, we introduce S$^3$M-Net, a novel joint learning framework developed to perform semantic segmentation and stereo matching simultaneously. Specifically, S$^3$M-Net shares the features extracted from RGB images between both tasks, resulting in an improved overall scene understanding capability. This feature sharing process is realized using a feature fusion adaption (FFA) module, which effectively transforms the shared features into semantic space and subsequently fuses them with the encoded disparity features. The entire joint learning framework is trained by minimizing a novel semantic consistency-guided (SCG) loss, which places emphasis on the structural consistency in both tasks. Extensive experimental results conducted on the vKITTI2 and KITTI datasets demonstrate the effectiveness of our proposed joint learning framework and its superior performance compared to other state-of-the-art single-task networks. Our project webpage is accessible at mias.group/S3M-Net.

Figures (7)

• Figure 1: The architecture of our proposed S$^3$M-Net for end-to-end joint learning of semantic segmentation and stereo matching.
• Figure 2: An illustration of our proposed FFA module.
• Figure 3: Qualitative experimental results of semantic segmentation on the vKITTI2 cabon2020vkitti2 dataset: (a) RGB images; (b)-(k) semantic segmentation results achieved by SegNet badrinarayanan2017segnet, U-Net ronneberger2015unet, PSPNet zhao2017pspnet, DeepLabv3+ chen2018deeplabv3plus, HRNet sun2019hrnet, BiSeNet V2 yu2021bisenetv2, Segmenter strudel2021segmenter, SegFormer xie2021segformer, Mask2Former cheng2022mask2former, and DDRNet hong2021ddrnet, respectively; (l)-(q) semantic segmentation results achieved by FuseNet hazirbas2017FuseNet, MFNet ha2017MFNet, RTFNet sun2019RTFNet, SNE-RoadSeg fan2020sneroadseg, OFF-Net min2022orfd, and RoadFormer li2023roadformer, respectively; (r)-(s) semantic segmentation results achieved by our proposed S$^3$M-Net w/o and w/ the use of the SCG loss, respectively; (t) ground truth annotations.
• Figure 4: Qualitative experimental results of semantic segmentation on the KITTI 2015 menze2015kitti dataset: (a) RGB images; (b)-(k) semantic segmentation results achieved by SegNet badrinarayanan2017segnet, U-Net ronneberger2015unet, PSPNet zhao2017pspnet, DeepLabv3+ chen2018deeplabv3plus, HRNet sun2019hrnet, BiSeNet V2 yu2021bisenetv2, Segmenter strudel2021segmenter, SegFormer xie2021segformer, Mask2Former cheng2022mask2former, and DDRNet hong2021ddrnet, respectively; (l)-(q) semantic segmentation results achieved by FuseNet hazirbas2017FuseNet, MFNet ha2017MFNet, RTFNet sun2019RTFNet, SNE-RoadSeg fan2020sneroadseg, OFF-Net min2022orfd, and RoadFormer li2023roadformer, respectively; (r)-(s) semantic segmentation results achieved by our proposed S$^3$M-Net w/o and w/ the use of the SCG loss, respectively; (t) ground truth annotations.
• Figure 5: Qualitative experimental results of stereo matching on the vKITTI2 cabon2020vkitti2 dataset: (a) left RGB images; (b)-(j) disparity maps estimated using PSMNet chang2018psmnet, GwcNet guo2019gwcnet, AANet xu2020aanet, LEA-Stereo cheng2020leastereo, RAFT-Stereo lipson2021raftstereo, CRE-Stereo li2022crestereo, ACVNet xu2022acvnet, PCWNetshen2022pcwnet, and IGEV-Stereo xu2023igevstereo, respectively; (k)-(l) disparity maps estimated using our proposed S$^3$M-Net w/o and w/ the use of the SCG loss, respectively.