GeoPurify: A Data-Efficient Geometric Distillation Framework for Open-Vocabulary 3D Segmentation

TL;DR

GeoPurify tackles open-vocabulary 3D segmentation by addressing the disconnect between rich 2D semantics and 3D geometry. It introduces a data-efficient teacher-student framework where a sparse 3D Student Affinity Network learns latent geometric affinities from unlabeled geometry via contrastive distillation of a frozen 3D SSL teacher, and a frozen 2D VLM projects multi-view images to create initial semantic 3D features that are then purified by geometry-guided pooling. The method demonstrates strong data efficiency, achieving competitive or superior results on ScanNetV2, Matterport3D, and ScanNet200 using only about $1.5\%$ of the training data, and shows robust cross-dataset generalization. The combination of segmentation-as-understanding, geometric priors, and efficient distillation offers a scalable pathway for annotation-free 3D perception with real-world impact in robotics and AR.

Abstract

Recent attempts to transfer features from 2D Vision-Language Models (VLMs) to 3D semantic segmentation expose a persistent trade-off. Directly projecting 2D features into 3D yields noisy and fragmented predictions, whereas enforcing geometric coherence necessitates costly training pipelines and large-scale annotated 3D data. We argue that this limitation stems from the dominant segmentation-and-matching paradigm, which fails to reconcile 2D semantics with 3D geometric structure. The geometric cues are not eliminated during the 2D-to-3D transfer but remain latent within the noisy and view-aggregated features. To exploit this property, we propose GeoPurify that applies a small Student Affinity Network to purify 2D VLM-generated 3D point features using geometric priors distilled from a 3D self-supervised teacher model. During inference, we devise a Geometry-Guided Pooling module to further denoise the point cloud and ensure the semantic and structural consistency. Benefiting from latent geometric information and the learned affinity network, GeoPurify effectively mitigates the trade-off and achieves superior data efficiency. Extensive experiments on major 3D benchmarks demonstrate that GeoPurify achieves or surpasses state-of-the-art performance while utilizing only about 1.5% of the training data. Our codes and checkpoints are available at [https://github.com/tj12323/GeoPurify](https://github.com/tj12323/GeoPurify).

Figures (6)

• Figure 1: The Fundamental Disconnect: Semantic Richness vs. Geometric Coherence.Left: Original RGB 3D scene. Middle: Features distilled from 2D VLMs zou2023generalized offer rich semantics but exhibit geometric inconsistency, leading to fragmented and noisy segmentations. Right: Features from 3D self-supervised model wu2025sonata provide strong structural coherence with geometrically sound segments, yet inherently lack open-vocabulary semantic understanding. Our proposed method aims to bridge this critical gap by purifying the semantically rich 2D features with robust 3D geometric priors.
• Figure 2: GeoPurify: A Data-Efficient Pipeline for Geometric Purification of 3D Semantic Features. Our method consists of two stages. 1) Training (left, dotted path): A Student Affinity Network ($\phi_S$) is trained to comprehend 3D structure. It learns geometric relationships directly from the point cloud, using contrastive distillation to mimic the embeddings of a powerful, frozen 3D SSL teacher ($\phi_T$). This training phase requires no 3D semantic labels. 2) Inference (right, solid path): A frozen 2D VLM ($\Psi_{2D}$) generates initial 3D features by projecting rich semantic content from multi-view images. These features, however, are geometrically inconsistent. The pre-trained student network then applies a geometry-aware pooling, using its learned affinities to refine the initial features. This process yields a final representation that is both semantically rich and geometrically coherent.
• Figure 3: The Geometry-Guided Pooling Mechanism.Left: Initial point cloud features aggregated from 2D views exhibit significant fragmentation and geometric inconsistency. Center: Our pooling mechanism operates locally. For a given query point, the learned affinity matrix identifies nearby points on the same surface as having high affinity (green lines) while assigning low affinity to points on other surfaces (gray lines). Right: By iteratively averaging features among high-affinity neighbors, the process denoises the initial features, producing a final output that is both semantically rich and geometrically coherent.
• Figure 4: Visualization of Feature Coherence.Point clouds are colored by applying k-means clustering to their features. Top Row (Initial Features): Features aggregated directly from 2D views lead to fragmented and geometrically inconsistent clusters. Bottom Row (Purified Features): Our Geometry-Guided Pooling produces smooth, coherent features that align with the underlying 3D structure.
• Figure 5: Qualitative Comparison on ScanNetV2 and Matterport3D.Visual results on challenging indoor scenes. From top to bottom: Input RGB point cloud, Ground Truth segmentation, Baseline predictions (from aggregated 2D features), and our GeoPurify predictions. Our method generates significantly cleaner segmentation maps with sharper boundaries and more complete object segments compared to the baseline.