SPNeRF: Open Vocabulary 3D Neural Scene Segmentation with Superpoints

TL;DR

SPNeRF addresses the challenge of zero-shot open-vocabulary 3D segmentation by integrating CLIP-based language features with a NeRF representation guided by geometric primitives. It introduces primitive-wise CLIP embeddings and a primitive merging block with affinity scores to align semantic labels with 3D geometry, without training additional segmentation models. The approach extends LERF with consistency and density losses and a progressive training schedule, achieving a $17.25\%$ mIoU and $31.07\%$ mAcc on Replica, improving LERF by $6.75$ percentage points in mIoU and remaining competitive with OpenNeRF in mAcc. This yields a zero-shot 3D segmentation framework that preserves CLIP's broad language capabilities while improving spatial coherence, offering practical benefits for open-world perception in robotics and autonomous systems.

Abstract

Open-vocabulary segmentation, powered by large visual-language models like CLIP, has expanded 2D segmentation capabilities beyond fixed classes predefined by the dataset, enabling zero-shot understanding across diverse scenes. Extending these capabilities to 3D segmentation introduces challenges, as CLIP's image-based embeddings often lack the geometric detail necessary for 3D scene segmentation. Recent methods tend to address this by introducing additional segmentation models or replacing CLIP with variations trained on segmentation data, which lead to redundancy or loss on CLIP's general language capabilities. To overcome this limitation, we introduce SPNeRF, a NeRF based zero-shot 3D segmentation approach that leverages geometric priors. We integrate geometric primitives derived from the 3D scene into NeRF training to produce primitive-wise CLIP features, avoiding the ambiguity of point-wise features. Additionally, we propose a primitive-based merging mechanism enhanced with affinity scores. Without relying on additional segmentation models, our method further explores CLIP's capability for 3D segmentation and achieves notable improvements over original LERF.

Figures (4)

• Figure 1: Overview of SPNeRF Pipeline. Given 2D posed images as input, SPNeRF optimizes a 3D CLIP feature field by distilling vision-language embeddings from the CLIP image encoder. Simultaneously, the radiance field is trained in parallel. Superpoints, which are extracted from the 3D geometry, are used to enhance both the radiance field and CLIP feature field during optimization, ensuring better alignment of semantic and spatial information. The training process leverages a combination of loss functions $\mathcal{L}$ to refine the consistency and accuracy of the feature representations. The merging block combines query labels with superpoints information to produce semantic segmentation results.
• Figure 2: 3D segmentation results in comparison to other methods. Qualitative comparison of 3D semantic segmentation results on the Replica dataset. Rows display results from (top to bottom) ground truth, LERF, OpenNeRF and SPNeRF, across 3 indoor scenes. SPNeRF demonstrates improved boundary coherence and segmentation accuracy in general.
• Figure 3: Ablation comparison of consistency loss. Even contrained by the fragmented superpoints, the results w/o loss tend to be consistent due to the image embedding characteristic of CLIP. The consistency loss helps the model to get more precise semantic info, especially for large superpoints like wall surfaces.
• Figure 4: Ablation comparison. The figure illustrates the improvement by consistency loss and affinity score.