COS3D: Collaborative Open-Vocabulary 3D Segmentation

TL;DR

COS3D tackles open-vocabulary 3D segmentation by introducing a collaborative field that jointly leverages instance (segmentation-aware) and language (text-aligned) cues. It learns an instance-to-language mapping to build a meaningful language field during training, and uses adaptive language-to-instance prompt refinement during inference to produce high-quality 3D segmentations from text prompts. The method achieves state-of-the-art results on LeRF and ScanNetv2, while delivering improved training efficiency and supporting versatile applications such as novel image-based segmentation, explicit hierarchical queries, and robotics. By integrating 3D-Gaussian Splatting, 2D foundation models (SAM/CLIP), and a two-stage learning strategy, COS3D offers a practical, scalable solution for open-vocabulary 3D perception in real-world scenarios.

Abstract

Open-vocabulary 3D segmentation is a fundamental yet challenging task, requiring a mutual understanding of both segmentation and language. However, existing Gaussian-splatting-based methods rely either on a single 3D language field, leading to inferior segmentation, or on pre-computed class-agnostic segmentations, suffering from error accumulation. To address these limitations, we present COS3D, a new collaborative prompt-segmentation framework that contributes to effectively integrating complementary language and segmentation cues throughout its entire pipeline. We first introduce the new concept of collaborative field, comprising an instance field and a language field, as the cornerstone for collaboration. During training, to effectively construct the collaborative field, our key idea is to capture the intrinsic relationship between the instance field and language field, through a novel instance-to-language feature mapping and designing an efficient two-stage training strategy. During inference, to bridge distinct characteristics of the two fields, we further design an adaptive language-to-instance prompt refinement, promoting high-quality prompt-segmentation inference. Extensive experiments not only demonstrate COS3D's leading performance over existing methods on two widely-used benchmarks but also show its high potential to various applications,~\ie, novel image-based 3D segmentation, hierarchical segmentation, and robotics. The code is publicly available at \href{https://github.com/Runsong123/COS3D}{https://github.com/Runsong123/COS3D}.

Figures (6)

• Figure 1: Comparing different paradigms. (a) Language-based methods kerr2023lerfqin2024langsplatShi_2024_CVPRjun2025drji2025fastlgs directly learn a 3D language field for open-vocabulary segmentation. (b) Segmentation-based methods ye2023gaussianwu2024opengaussianli2024instancegaussian perform class-agnostic segmentation then post selection. (c) COS3D addresses existing limitations with a new collaborative prompt-segmentation framework that connects knowledge from the language and instance fields in the training and inference. The solid line (dotted line) indicates inference (training). (d) COS3D clearly outperforms existing methods on both segmentation quality and training efficiency. Note that "fast version" refers to a result of our approach under a short optimization time setting (see Sec.\ref{['sec:ablation']}), while the time and performance of the baselines are sourced from their publications jun2025drwu2024opengaussian.
• Figure 2: Overview of our proposed COS3D method. We first introduce collaborative field, comprising an instance field and a language field (see Sec.\ref{['sec:collaborative']}). During training, we first learn the instance field to encode instance information and propose a novel instance-to-language (Ins2Lang) mapping to construct the language field (see Sec.\ref{['sec:learning']}). During inference, leveraging the 3D relevance map from the language field, we design an adaptive language-to-instance (Lang2Ins) prompt refinement to further guide the instance field, enabling accurate segmentation (see Sec.\ref{['sec:inference']}).
• Figure 3: Visual comparisons on LeRF kerr2023lerf.
• Figure 4: Open-vocabulary 3D Gaussian segmentation on the LeRF dataset. Our method outperforms previous open-sourced SOTA methods (i.e., LangSplat, OpenGaussian) in accurately identifying the 3D objects corresponding to text queries with fewer artifacts. Here, we present our results using kernel regression for visual comparison, and more results are provided in Supp.
• Figure 5: Open-vocabulary Gaussian segmentation on ScanNetv2 dai2017scannet dataset. Our method outperforms the previous open-sourced SOTA approach (i.e., OpenGaussian) in accurately identifying 3D objects for various text queries. In addition, we use green boxes to indicate regions of accurate predictions and red boxes to indicate regions of missing predictions. Here, we present our results using kernel regression for visual comparison, and more results are provided in Supp.