3D Can Be Explored In 2D: Pseudo-Label Generation for LiDAR Point Clouds Using Sensor-Intensity-Based 2D Semantic Segmentation

TL;DR

This work tackles 3D LiDAR semantic segmentation by eliminating the need for 3D annotations and inference-time camera data. It builds a pipeline that colors LiDAR points by sensor intensity, generates numerous 2D views from virtual camera poses, and applies a pretrained 2D segmentation model to produce 3D pseudo-labels via back-projection and vote aggregation. The authors demonstrate the approach on nuScenes, perform extensive ablations on view generation and fusion strategies, and show competitive results when these pseudo-labels are used for unsupervised domain adaptation with SemanticKITTI as the source. The study highlights the potential of 2D-to-3D label propagation to reduce labeling effort while enabling robust 3D segmentation across domains, albeit with challenges in occlusion and dynamic scenes that warrant future work.

Abstract

Semantic segmentation of 3D LiDAR point clouds, essential for autonomous driving and infrastructure management, is best achieved by supervised learning, which demands extensive annotated datasets and faces the problem of domain shifts. We introduce a new 3D semantic segmentation pipeline that leverages aligned scenes and state-of-the-art 2D segmentation methods, avoiding the need for direct 3D annotation or reliance on additional modalities such as camera images at inference time. Our approach generates 2D views from LiDAR scans colored by sensor intensity and applies 2D semantic segmentation to these views using a camera-domain pretrained model. The segmented 2D outputs are then back-projected onto the 3D points, with a simple voting-based estimator that merges the labels associated to each 3D point. Our main contribution is a global pipeline for 3D semantic segmentation requiring no prior 3D annotation and not other modality for inference, which can be used for pseudo-label generation. We conduct a thorough ablation study and demonstrate the potential of the generated pseudo-labels for the Unsupervised Domain Adaptation task.

Figures (2)

• Figure 1: Pipeline. As a preliminary step, a 2D semantic segmentation model is trained on augmented camera images. An input sequence of LiDAR 3D scans and sensor poses is aligned and colorized using sensor intensity. A large number of 2D greyscale images are then generated from virtual camera poses along the sensor's trajectory. The 2D model is applied to obtain segmentation logits and labels for each view which are then projected back onto the 3D points. These votes are counted and each point is assigned a class based on the chosen election estimator.
• Figure 2: Comparison of the pseudo-labels generated on the nuScenes dataset caesar_nuscenes_2020. IoU per class and mIoU for all annotated scenes. Transformations such as cropping are applied to obtain a relevant comparison.