SAM-guided Pseudo Label Enhancement for Multi-modal 3D Semantic Segmentation

TL;DR

This work tackles domain shift in multi-modal 3D semantic segmentation by densifying high-quality pseudo-labels through 2D priors from the Segment Anything Model (SAM). It introduces Mask Label Assignment (MLA) to derive per-mask labels with strict size, purity, and representativity constraints, followed by Geometry-Aware Progressive Propagation (GAPP) to safely spread labels within SAM masks while mitigating 2D-3D misalignment. Empirical results across multiple cross-domain benchmarks and adaptation settings show significantly denser pseudo-labels and improved mIoU over strong baselines, with ablations confirming the contributions of both MF constraints and GAPP. The method is applicable to both unsupervised and source-free domain adaptation, offering a practical path to robust multi-modal 3D semantic segmentation in real-world deployment.

Abstract

Multi-modal 3D semantic segmentation is vital for applications such as autonomous driving and virtual reality (VR). To effectively deploy these models in real-world scenarios, it is essential to employ cross-domain adaptation techniques that bridge the gap between training data and real-world data. Recently, self-training with pseudo-labels has emerged as a predominant method for cross-domain adaptation in multi-modal 3D semantic segmentation. However, generating reliable pseudo-labels necessitates stringent constraints, which often result in sparse pseudo-labels after pruning. This sparsity can potentially hinder performance improvement during the adaptation process. We propose an image-guided pseudo-label enhancement approach that leverages the complementary 2D prior knowledge from the Segment Anything Model (SAM) to introduce more reliable pseudo-labels, thereby boosting domain adaptation performance. Specifically, given a 3D point cloud and the SAM masks from its paired image data, we collect all 3D points covered by each SAM mask that potentially belong to the same object. Then our method refines the pseudo-labels within each SAM mask in two steps. First, we determine the class label for each mask using majority voting and employ various constraints to filter out unreliable mask labels. Next, we introduce Geometry-Aware Progressive Propagation (GAPP) which propagates the mask label to all 3D points within the SAM mask while avoiding outliers caused by 2D-3D misalignment. Experiments conducted across multiple datasets and domain adaptation scenarios demonstrate that our proposed method significantly increases the quantity of high-quality pseudo-labels and enhances the adaptation performance over baseline methods.

Figures (4)

• Figure 1: Overview of the proposed method. The initial pseudo-labels tend to be accurate but sparse. Our proposed method aims to enhance the pseudo-labels by utilizing the grouping information of SAM masks.
• Figure 2: Visual explanation of the proposed method. The SAM masks are sorted in descending order based on their area. For $M_1$, the pseudo-labels contain two main classes (yellow and green) and violate the purity constraint in Equation \ref{['equ:rp']}. Thus the mask label is not valid and there is no update to the pseudo-labels. For $M_2$ and $M_{N'}$, MLA generates valid mask labels and we apply GAPP to enhance the pseudo-labels within the mask. Then we update the pseudo-labels $\hat{y}$ and proceed to the next mask.
• Figure 3: Visualization of the proposed GAPP. Note the green points on the Ground Truth images (rightmost) belong to the background, not the car. The lidar paths of these green points are occluded in the camera image, thus they 'appear' to be on the car due to the sensor location mismatch between the lidar and camera. Unlike Direct Propagation (DP), GAPP successfully avoids those incorrect mask labels assigned to the car.
• Figure 4: Visualization of the SAM masks and enhanced pseudo-labels. The proposed method significantly improves the pseudo-label quality.