3D Weakly Supervised Semantic Segmentation via Class-Aware and Geometry-Guided Pseudo-Label Refinement

TL;DR

The paper tackles 3D weakly supervised semantic segmentation using scene-level labels by addressing pseudo-label quality and the underutilization of 3D geometry. It introduces a two-stage framework: first, Class-Aware Label Refinement (CALR) and Geometry-Aware Label Refinement (GALR) generate high-quality, balanced pseudo labels by combining semantic cues with 3D geometric priors; second, Self-Training with Label Propagation (STLP) propagates labels to unlabeled regions through iterative refinement. Empirical results on ScanNet and S3DIS show state-of-the-art performance under weak supervision and competitive performance in unsupervised settings, highlighting the method's robustness and practical value. The work demonstrates that integrating 3D geometric priors with class-aware refinement and label propagation can substantially reduce annotation costs while preserving high segmentation accuracy.

Abstract

3D weakly supervised semantic segmentation (3D WSSS) aims to achieve semantic segmentation by leveraging sparse or low-cost annotated data, significantly reducing reliance on dense point-wise annotations. Previous works mainly employ class activation maps or pre-trained vision-language models to address this challenge. However, the low quality of pseudo-labels and the insufficient exploitation of 3D geometric priors jointly create significant technical bottlenecks in developing high-performance 3D WSSS models. In this paper, we propose a simple yet effective 3D weakly supervised semantic segmentation method that integrates 3D geometric priors into a class-aware guidance mechanism to generate high-fidelity pseudo labels. Concretely, our designed methodology first employs Class-Aware Label Refinement module to generate more balanced and accurate pseudo labels for semantic categrories. This initial refinement stage focuses on enhancing label quality through category-specific optimization. Subsequently, the Geometry-Aware Label Refinement component is developed, which strategically integrates implicit 3D geometric constraints to effectively filter out low-confidence pseudo labels that fail to comply with geometric plausibility. Moreover, to address the challenge of extensive unlabeled regions, we propose a Label Update strategy that integrates Self-Training to propagate labels into these areas. This iterative process continuously enhances pseudo-label quality while expanding label coverage, ultimately fostering the development of high-performance 3D WSSS models. Comprehensive experimental validation reveals that our proposed methodology achieves state-of-the-art performance on both ScanNet and S3DIS benchmarks while demonstrating remarkable generalization capability in unsupervised settings, maintaining competitive accuracy through its robust design.

Figures (10)

• Figure 1: Comparison between previous 3D WSSS methods and our proposed approach. (a) Pseudo label analysis; (b) The typical pipeline of previous SOTA methods; (c) The workflow of our proposed methodology.
• Figure 2: The proposed Pseudo Label Generation and Refinement procedures. We first extract 2D embeddings $F_{2D}$ and text embeddings $F_C$ using a pretrained VLM. The 2D embeddings are back-projected via camera calibration to obtain 2D-projected embeddings $P_{2D}$. Prediction logits are computed by multiplying $F_C$ and $P_{2D}$, then filtered with the scene-level label mask $M$. After ranking, initial pseudo labels ${Y^0_p}$ and confidence scores ${R^0}$ are obtained. CALR selects top-$V$% pseudo labels per category based on $R^0$ to ensure class balance and confidence. GALR refines labels by superpoint overlap: if a dominant category in a superpoint exceeds a threshold, the block is assigned that category; otherwise, it remains unlabeled. The final pseudo labels $Y^0$ are obtained after refinement. Notably, in $R^0$, color depth represents confidence, akin to a heatmap, where darker colors indicate higher confidence. Points with the same color in the pseudo labels $Y^0$ correspond to the same predicted category, and the dotted circle in $Q$ denotes points within the same superpoint.
• Figure 3: We present the performance of pseudo labels across different confidence intervals, along with the proportion of all points within each changed confidence range. The results indicate that higher confidence levels correspond to better segmentation performance. Notably, more than half of the pseudo labels exhibit confidence below 9%, highlighting the ambiguity of the original pseudo labels.
• Figure 4: Visualization of applying global of top-$V$% selection (Base), our CALR and GALR on ScanNet dataset. From left to right: ground truth, global top-$V$%, our CALR results and GALR results.
• Figure 5: The proposed Self-Training with Label Propagation follows an iterative approach. First, the model is trained using pseudo labels $Y^T$ from the previous step. Then, inference on the training set generates updated predictions $Y^{T+1}_p$ and confidence scores $R^{T+1}$. To propagate pseudo labels to unlabeled regions, the Label Update strategy retains previous pseudo labels $Y^T$ while incorporating reliable new predictions to generate updated pseudo labels $Y^{T+1}_u$. GALR further refines them to obtain the final updated pseudo labels $Y^{T+1}$. This process iterates, using updated pseudo labels as supervision, progressively extending labels to unlabeled regions. Notably, in $R^{T+1}$, color depth represents confidence levels. Points with the same color in the pseudo labels $Y^T$ and $Y_p^{T+1}$ correspond to the same predicted category, while black-colored points denote masked regions that do not require any updates.