SegDINO3D: 3D Instance Segmentation Empowered by Both Image-Level and Object-Level 2D Features

TL;DR

SegDINO3D addresses 3D instance segmentation under limited 3D data by integrating rich 2D cues from pretrained detectors into a DETR-like 3D transformer. It enriches 3D point features with image-level information via Nearest View Sampling and fuses them with a 3D encoder, while decoding uses 3D box queries and compact 2D object queries with distance-aware and box-modulated attention to refine masks. The method achieves state-of-the-art results on ScanNetv2 and the challenging ScanNet200 benchmarks, with notably faster convergence and strong performance on long-tail classes. These contributions demonstrate effective cross-modal fusion and efficient cross-attention strategies for 3D scene understanding.

Abstract

In this paper, we present SegDINO3D, a novel Transformer encoder-decoder framework for 3D instance segmentation. As 3D training data is generally not as sufficient as 2D training images, SegDINO3D is designed to fully leverage 2D representation from a pre-trained 2D detection model, including both image-level and object-level features, for improving 3D representation. SegDINO3D takes both a point cloud and its associated 2D images as input. In the encoder stage, it first enriches each 3D point by retrieving 2D image features from its corresponding image views and then leverages a 3D encoder for 3D context fusion. In the decoder stage, it formulates 3D object queries as 3D anchor boxes and performs cross-attention from 3D queries to 2D object queries obtained from 2D images using the 2D detection model. These 2D object queries serve as a compact object-level representation of 2D images, effectively avoiding the challenge of keeping thousands of image feature maps in the memory while faithfully preserving the knowledge of the pre-trained 2D model. The introducing of 3D box queries also enables the model to modulate cross-attention using the predicted boxes for more precise querying. SegDINO3D achieves the state-of-the-art performance on the ScanNetV2 and ScanNet200 3D instance segmentation benchmarks. Notably, on the challenging ScanNet200 dataset, SegDINO3D significantly outperforms prior methods by +8.6 and +6.8 mAP on the validation and hidden test sets, respectively, demonstrating its superiority.

Figures (5)

• Figure 1: (a) The dashed lines indicate different ways of utilizing 2D image features. "2D-3D Mapping" denotes the association between 3D points and 2D features, and "$\oplus$" indicates feature fusion. (b) SegDINO3D obtains significantly better performance and faster convergence speed.
• Figure 2: (a) Overview of SegDINO3D, our contributions are highlighted in green. (b) Visual illustration of the Nearest View Sampling operation. Each 3D point calculates its distance to all the views that it is visible to finds the top-k nearest views (red dash lines). (c) Visual illustration of 2D object queries construction. Each 2D object query is assigned a 3D center computed as the medoid of its corresponding 3D points, which are derived by projecting the depth values within its 2D mask using the camera parameters. The green points on the right side show the distribution of the 2D object queries' 3D centers in the scene.
• Figure 3: Visual comparison with the baseline method on the ScanNet200 validation set. We visualize only the predictions with a confidence score greater than 0.5. The red boxes indicate instances missed by the baseline, the blue boxes indicate instances incorrectly segmented by the baseline, and the green boxes indicate additional instances segmented by our method.
• Figure 4: Visualization of the box-modulation's effect. (a) From top to bottom, we present an example RGB image from the input images and the ground‑truth segmentation results of this scene. (b) Shows the positional similarity within the box‑modulated cross‑attention module of our method. The green point indicates the position of the current query, while the red and blue points represent the similarity between each superpoint and the query. Red indicates high similarity, and blue indicates low similarity. Regions with relatively high similarity are highlighted with green boxes. Here, we present examples of a radiator and a chair, shown from top to bottom, respectively. (c) Depicts the positional similarity without applying box modulation.
• Figure 5: Visualization of the predicted 3D instance bounding boxes and 3D instance masks.