OpenInsGaussian: Open-vocabulary Instance Gaussian Segmentation with Context-aware Cross-view Fusion

TL;DR

OpenInsGaussian tackles open-vocabulary semantic understanding for 3D Gaussian Splatting by addressing context loss from maskcropping and cross-view inconsistencies. It introduces Context-Aware Feature Extraction (CLIP intermediate-feature cropping) and Attention-Driven Feature Aggregation (similarity-based multi-view fusion) to bind language to 3D Gaussians. The method combines SAM-based 3D mask initialization, context-rich language embeddings, two-level codebook discretization, and robust 3D–2D association, yielding state-of-the-art results on ScanNet and LeRF benchmarks. The work advances practical 3D scene understanding by delivering robust, open-vocabulary segmentation with improved efficiency and cross-view consistency.

Abstract

Understanding 3D scenes is pivotal for autonomous driving, robotics, and augmented reality. Recent semantic Gaussian Splatting approaches leverage large-scale 2D vision models to project 2D semantic features onto 3D scenes. However, they suffer from two major limitations: (1) insufficient contextual cues for individual masks during preprocessing and (2) inconsistencies and missing details when fusing multi-view features from these 2D models. In this paper, we introduce \textbf{OpenInsGaussian}, an \textbf{Open}-vocabulary \textbf{Ins}tance \textbf{Gaussian} segmentation framework with Context-aware Cross-view Fusion. Our method consists of two modules: Context-Aware Feature Extraction, which augments each mask with rich semantic context, and Attention-Driven Feature Aggregation, which selectively fuses multi-view features to mitigate alignment errors and incompleteness. Through extensive experiments on benchmark datasets, OpenInsGaussian achieves state-of-the-art results in open-vocabulary 3D Gaussian segmentation, outperforming existing baselines by a large margin. These findings underscore the robustness and generality of our proposed approach, marking a significant step forward in 3D scene understanding and its practical deployment across diverse real-world scenarios.

Figures (6)

• Figure 1: We introduce OpenInsGaussian, a simple yet effective approach that improves semantic understanding in 3D Gaussian segmentation. By leveraging Context-Aware Feature Extraction and Multi-View Attention, OpenInsGaussian addresses two commonly overlooked issues in existing language understanding pipelines: the loss of contextual information when extracting features from cropped masks and the inconsistencies introduced by multi-view fusion.
• Figure 2: Given a set of images and pretrained 3D Gaussians, our pipeline consists of four key steps: (1) We preprocess the input images using SAM saga to obtain object masks and fuse local and context-aware CLIP clip encoded language features to generate context-aware language embeddings for each mask. (2) The SAM masks are then used to train class-agnostic instance features for each 3D Gaussian. (3) We discretize the 3D Gaussians by clustering instance features using a hierarchical approach. (4) Finally, we employ an attention-driven feature aggregation method to associate language embeddings with the segmented instances, enabling open-vocabulary understanding.
• Figure 3: Example of a cropped image patch in local feature extraction on Scannet dataset.
• Figure 4: Example of language embedding with different feature extraction strategies on Scannet dataset.
• Figure 5: Open-vocabulary 3D point cloud understanding on the Scannet dataset.