SegPoint: Segment Any Point Cloud via Large Language Model

TL;DR

SegPoint presents a unified framework that uses a multi-modal LLM to perform point-level segmentation across 4 3D tasks, leveraging a Geometric Enhancer Module and Geometric-guided Feature Propagation to inject local geometry and preserve detail. A new Instruct3D benchmark with 2,565 instruction–point cloud pairs evaluates reasoning-driven segmentation, complementing established 3D benchmarks. Empirical results show competitive performance on ScanRefer and ScanNet and notable gains on Instruct3D, validating the approach’s versatility and reasoning capabilities. The work advances language-guided perception in 3D and points to future work on extending prompts beyond text prompts via a prompt encoder.

Abstract

Despite significant progress in 3D point cloud segmentation, existing methods primarily address specific tasks and depend on explicit instructions to identify targets, lacking the capability to infer and understand implicit user intentions in a unified framework. In this work, we propose a model, called SegPoint, that leverages the reasoning capabilities of a multi-modal Large Language Model (LLM) to produce point-wise segmentation masks across a diverse range of tasks: 1) 3D instruction segmentation, 2) 3D referring segmentation, 3) 3D semantic segmentation, and 4) 3D open-vocabulary semantic segmentation. To advance 3D instruction research, we introduce a new benchmark, Instruct3D, designed to evaluate segmentation performance from complex and implicit instructional texts, featuring 2,565 point cloud-instruction pairs. Our experimental results demonstrate that SegPoint achieves competitive performance on established benchmarks such as ScanRefer for referring segmentation and ScanNet for semantic segmentation, while delivering outstanding outcomes on the Instruct3D dataset. To our knowledge, SegPoint is the first model to address these varied segmentation tasks within a single framework, achieving satisfactory performance.

Figures (5)

• Figure 1: Example of functionality in SegPoint. SegPoint can complete various point cloud tasks in a unified framework by leveraging task-specific prompts, including 1) 3D instruction segmentation, 2) 3D referring segmentation, 3) 3D semantic segmentation, and 4) 3D open-vocabulary semantic segmentation.
• Figure 2: The pipeline of SegPoint. Given input point cloud and text query, the multi-modal LLM $\mathcal{F}$ generates text output. Geometric Enhancer Module $\mathcal{G}$ injects geometric information into Point Encoder $\mathcal{E}$ and obtains point features $\hat{\hbox{\boldmath$f$}}_{point}$. Per-point embeddings ${\hbox{\boldmath$f$}}_{\mathcal{P}}$ derived from Geometric-guided Feature Propagation $\mathcal{P}$ multiplied with the embedding associated with the <SEG> token yield the final segmentation masks.
• Figure 3: Architecture of the proposed (b) Geometric Enhance Module (GEM) and (c) Geometric-guided Feature Propagation (GFP) interaction with (a) Point Encoder.
• Figure 4: (Best viewed in color) We visualize the feature responses between a given point (in red) and other points in the scene from per-point embeddings ${\hbox{\boldmath$f$}}_{\mathcal{P}}$ for the baseline and our SegPoint, respectively. The color changes from yellow to red, indicating increasing feature similarity.
• Figure 5: Qualitative results from val split of Instruct3D. SegPoint understand the human instruction and accurately segment the target object. We omitted the "please output segmentation mask" in the sentence for simplicity.