Multimodal Point Cloud Semantic Segmentation With Virtual Point Enhancement

TL;DR

This paper tackles the challenge of sparse and irregular LiDAR point clouds in semantic segmentation by introducing Virtual Point Enhancement (VPE), which augments LiDAR data with dense, image-derived virtual points. A Spatial Difference-driven Adaptive Filtering Module (SDAFM) selectively retains reliable virtual points based on density and distance, mitigating noise and computational burden, while a Noise-robust Sparse Feature Encoder (NSFE) performs robust 2D-3D feature extraction and a fine-grained enhancement path to improve sparse geometric representations. The approach uses sparse supervision with voxel-level labels and combines cross-entropy with Lovász-softmax losses, achieving notable gains such as an improvement of 2.89 percentage points in mIoU on nuScenes when 7.7% virtual points are introduced. Overall, VPENet demonstrates strong performance on SemanticKITTI and nuScenes, offering a practical multimodal fusion strategy that alleviates LiDAR sparsity and enhances segmentation for medium-range and small targets, with potential applicability to broader autonomous-driving tasks like 3D reconstruction and semantic completion.

Abstract

LiDAR-based 3D point cloud recognition has been proven beneficial in various applications. However, the sparsity and varying density pose a significant challenge in capturing intricate details of objects, particularly for medium-range and small targets. Therefore, we propose a multi-modal point cloud semantic segmentation method based on Virtual Point Enhancement (VPE), which integrates virtual points generated from images to address these issues. These virtual points are dense but noisy, and directly incorporating them can increase computational burden and degrade performance. Therefore, we introduce a spatial difference-driven adaptive filtering module that selectively extracts valuable pseudo points from these virtual points based on density and distance, enhancing the density of medium-range targets. Subsequently, we propose a noise-robust sparse feature encoder that incorporates noise-robust feature extraction and fine-grained feature enhancement. Noise-robust feature extraction exploits the 2D image space to reduce the impact of noisy points, while fine-grained feature enhancement boosts sparse geometric features through inner-voxel neighborhood point aggregation and downsampled voxel aggregation. The results on the SemanticKITTI and nuScenes, two large-scale benchmark data sets, have validated effectiveness, significantly improving 2.89\% mIoU with the introduction of 7.7\% virtual points on nuScenes.

Figures (8)

• Figure 1: The sparsity of point clouds. The left case is from the SemanticKITTI dataset and the right case is from the NuScenes dataset. As distance increases, LiDAR points become sparser, making it challenging to identify medium-range targets such as cars (marked with red boxes) and small targets with only a few points, such as pedestrians and bicycles (marked with green circles).
• Figure 2: Overview of our Virtual Point Enhancement (VPE) for Multi-modal Point Cloud Semantic Segmentation. It first through depth completion to obtain the dense but noisy virtual points from the original camera images. These points (in blue) are then merged with the original points (in red) and fed into the spatial difference-driven adaptive filtering module to increase the point cloud density of medium-range and small targets. Subsequently, the merged point cloud undergoes noise reduction through noise-robust feature extraction and enhancement with sparse geometric features, generating input for the next stage of the noise-robust sparse feature encoder. Finally, the co-visible point sparse supervision will be incorporated as an auxiliary loss to complement the primary multi-layer sparse supervision in the fine-grained feature enhancement.
• Figure 3: Noise in depth completion and adaptive filtering. (a) Images. (b) Combination of the original point with the depth-completion point (darker in color). (c) Combination of the original point with the adaptively filtered point (darker in color).
• Figure 4: The internal structure of noise-robust feature extraction. In each level, we perform sparse submanifold convolutions in both 2D image space and 3D point cloud space and dynamically adjust the feature weights, enhancing feature representation.
• Figure 5: Internal structure of the fine-grained feature enhancement (see Algorithm \ref{['alg:point_voxel_aggregate_encoder']}) which consists of the point feature aggregate and voxel feature aggregation. Point feature aggregation treats the neighboring voxels as points and aggregates point features into each point of the central voxel. Voxel feature aggregation downsamples the voxels at the current scale, extracts features through an MLP and finally indexes the features back to the current point.