vFusedSeg3D: 3rd Place Solution for 2024 Waymo Open Dataset Challenge in Semantic Segmentation

TL;DR

VFusedSeg3D tackles 3D semantic segmentation by fusing LiDAR geometry with camera semantics through a dual-stream architecture and multi-stage fusion modules (GFFM, SFAM, SFFM). The approach achieves a state-of-the-art validation mIoU of $72.46$% on Waymo Open Dataset, surpassing the LiDAR-base ($70.51$%) and PTv3 ($71.3$%) baselines. A resource-conscious training strategy, involving sequential training of the LiDAR, image, and fusion components on a single GPU, enables robust multi-modal integration with practical compute demands. These results demonstrate the practicality and effectiveness of cross-modal fusion for accurate and robust 3D perception in real-world settings.

Abstract

In this technical study, we introduce VFusedSeg3D, an innovative multi-modal fusion system created by the VisionRD team that combines camera and LiDAR data to significantly enhance the accuracy of 3D perception. VFusedSeg3D uses the rich semantic content of the camera pictures and the accurate depth sensing of LiDAR to generate a strong and comprehensive environmental understanding, addressing the constraints inherent in each modality. Through a carefully thought-out network architecture that aligns and merges these information at different stages, our novel feature fusion technique combines geometric features from LiDAR point clouds with semantic features from camera images. With the use of multi-modality techniques, performance has significantly improved, yielding a state-of-the-art mIoU of 72.46% on the validation set as opposed to the prior 70.51%.VFusedSeg3D sets a new benchmark in 3D segmentation accuracy. making it an ideal solution for applications requiring precise environmental perception.

Figures (2)

• Figure 1: vFusedSeg3d architecture
• Figure 2: Lidar and camera feature fusion: Geometric Feature Fusion Module and Semantic Feature Fusion Modules as designed by mseg3d_cvpr2023 but with some modifications