BEEP3D: Box-Supervised End-to-End Pseudo-Mask Generation for 3D Instance Segmentation

TL;DR

BEEP3D tackles the high cost of point-level annotations in 3D instance segmentation by enabling end-to-end training with box supervision. It uses a student-teacher EMA framework in which the teacher generates pseudo-masks guided by instance centers, while the student learns from both real and pseudo labels; two consistency losses align content and masked features to the teacher. The instance-center refinement and losses yield improved pseudo-mask quality and segmentation accuracy, achieving state-of-the-art results among box-supervised methods on ScanNetV2 and strong performance on S3DIS, with notably reduced training complexity. This approach significantly lowers annotation effort while maintaining competitive accuracy, accelerating practical deployment in real-world 3D scene understanding tasks.

Abstract

3D instance segmentation is crucial for understanding complex 3D environments, yet fully supervised methods require dense point-level annotations, resulting in substantial annotation costs and labor overhead. To mitigate this, box-level annotations have been explored as a weaker but more scalable form of supervision. However, box annotations inherently introduce ambiguity in overlapping regions, making accurate point-to-instance assignment challenging. Recent methods address this ambiguity by generating pseudo-masks through training a dedicated pseudo-labeler in an additional training stage. However, such two-stage pipelines often increase overall training time and complexity, hinder end-to-end optimization. To overcome these challenges, we propose BEEP3D-Box-supervised End-to-End Pseudo-mask generation for 3D instance segmentation. BEEP3D adopts a student-teacher framework, where the teacher model serves as a pseudo-labeler and is updated by the student model via an Exponential Moving Average. To better guide the teacher model to generate precise pseudo-masks, we introduce an instance center-based query refinement that enhances position query localization and leverages features near instance centers. Additionally, we design two novel losses-query consistency loss and masked feature consistency loss-to align semantic and geometric signals between predictions and pseudo-masks. Extensive experiments on ScanNetV2 and S3DIS datasets demonstrate that BEEP3D achieves competitive or superior performance compared to state-of-the-art weakly supervised methods while remaining computationally efficient.

Figures (4)

• Figure 1: Comparison of pseudo-label generation pipelines. (a) GaPro gapro_ngo and (b) BSNet bsnet_lu illustrates separate pre-training stage for pseudo-label generation, whereas (c) BEEP3D (ours) generates pseudo-mask within the training loop. Dashed outlines indicate parameters frozen during training.
• Figure 2: Overview of BEEP3D. The student and teacher models share the same architecture, consisting of a 3D U-Net backbone and transformer decoder layers with position and content queries. The teacher model generates pseudo-masks for overlapping regions using instance centers and FPS, while the student is supervised by the union of pseudo-masks $\widehat{m}^u$ and ground-truth masks $m^l$ from box annotations.
• Figure 3: Visualization results on ScanNetV2 validation set. The first column shows the RGB input with ground-truth 3D bounding boxes (in yellow), and the second column presents the annotated instance masks. Highlighted regions (in circles) indicate overlapping areas, where box-supervised methods frequently struggle due to ambiguity in point-to-instance assignment.
• Figure A1: Qualitative comparison of model variants from the ablation study on ScanNetV2. Yellow boxes indicate ground-truth instance boxes, and circles highlight overlapping regions.