UniC-Lift: Unified 3D Instance Segmentation via Contrastive Learning

TL;DR

The paper addresses the challenge of inconsistent 2D multi-view segmentation when lifting to 3D by introducing UniC-Lift, a unified single-stage framework built on 3D Gaussian Splatting that assigns a learnable embedding $\boldsymbol{v} \in \mathbb{R}^d$ to each Gaussian primitive and decodes labels through an Embedding-to-Label process. It optimizes a joint objective comprising $\mathcal{L}_{rendering}$, $\mathcal{L}_{cluster}$, $\mathcal{L}_{triplet}$ (on a linear projection of embeddings), and $\mathcal{L}_{3D}$, with boundary hard-mining to stabilize learning; $d$ is set to 12 and the triplet/3D losses are activated after $15{,}000$ iterations. The approach yields faster training and improved metrics (e.g., PQ$^{scene}$ and $mIoU$) on ScanNet, Replica3D, and Messy-Rooms, compared to baselines like Contrastive-Lift and Panoptic-Lifting, while enabling practical downstream tasks such as object manipulation and extraction. By bypassing clustering-based post-processing, UniC-Lift achieves $O(n)$ per-view inference and direct label decoding, enabling scalable 3D instance segmentation in complex scenes. This work advances efficient, accurate 3D scene understanding with unified optimization and practical editing capabilities.

Abstract

3D Gaussian Splatting (3DGS) and Neural Radiance Fields (NeRF) have advanced novel-view synthesis. Recent methods extend multi-view 2D segmentation to 3D, enabling instance/semantic segmentation for better scene understanding. A key challenge is the inconsistency of 2D instance labels across views, leading to poor 3D predictions. Existing methods use a two-stage approach in which some rely on contrastive learning with hyperparameter-sensitive clustering, while others preprocess labels for consistency. We propose a unified framework that merges these steps, reducing training time and improving performance by introducing a learnable feature embedding for segmentation in Gaussian primitives. This embedding is then efficiently decoded into instance labels through a novel "Embedding-to-Label" process, effectively integrating the optimization. While this unified framework offers substantial benefits, we observed artifacts at the object boundaries. To address the object boundary issues, we propose hard-mining samples along these boundaries. However, directly applying hard mining to the feature embeddings proved unstable. Therefore, we apply a linear layer to the rasterized feature embeddings before calculating the triplet loss, which stabilizes training and significantly improves performance. Our method outperforms baselines qualitatively and quantitatively on the ScanNet, Replica3D, and Messy-Rooms datasets.

Figures (23)

• Figure 1: Our method takes a set of multi-view RGB images as input, and passes these images through a pre-trained instance/semantic segmentation method. Although accurate, segmentation methods for 2D images do not generate multi-view consistent segmentation masks. Observe how labels vary across the images. Our method formulates this as a clustering problem in 3D, utilizing feature-vectors in 3D space which are trained using contrastive losses. Our method generates consistent segmentation masks. We further motivate the necessity of this work in Fig. \ref{['fig:why_fig']}
• Figure 2: Previous methods are multi-stage: (a) some pre-process 2D masks before lifting them to 3D (e.g., DM-NeRF wang2022dm, Gaussian Grouping ye2023gaussian, and Panoptic-Lift kundu2022panoptic), while others use (b) a separate clustering algorithms on learned embeddings (e.g., Contrastive-Lift bhalgat2023contrastive). (c) In contrast, our method uses a single, unified representation to perform segmentation directly.
• Figure 3: Intuition of Embedding-To-Label Process. A toy experiment demonstrating how contrastive learning enables direct label prediction. Embeddings, initially distributed in a constrained space (e.g., $[0,1]^2$), converge to distinct corners during training. Each corner effectively becomes a unique binary code that can be thresholded and mapped to a label, removing the need for post-processing.
• Figure 4: Overview of our pipeline. Given input (a) multi-view RGB images and (c) inconsistent segmentation maps, we optimize (b) 3D representation to lift 2D segmentation labels to 3D. To learn the view-dependent changes associated with the novel-view synthesis task, we take a rendering loss: $\mathcal{L}_{rendering}$ with GT RGB image. For optimizing the learnable 3D vector embedding we apply $\mathcal{L}_{cluster}$ to the rasterized embedding and further apply $\mathcal{L}_{triplet}$ by passing it through a linear layer. Further, we apply a 3D loss $\mathcal{L}_{3D}$ to the primitives of 3D representation. More details are in Method section.
• Figure 5: Qualitative comparison of our method with Contrastive lift on scenes from ScanNet dataset. Regions, where Contrastive lift performs poorly, are highlighted with red boxes.