G2P: Gaussian-to-Point Attribute Alignment for Boundary-Aware 3D Semantic Segmentation

TL;DR

G2P addresses geometric bias in 3D point cloud segmentation by directly transferring appearance cues from 3D Gaussian Splatting to points while preserving original geometry. It introduces Gaussian-to-Point feature augmentation, scale-based boundary extraction, and Gaussian attribute-guided learning via appearance distillation from a self-supervised teacher, all operating in native 3D space without 2D supervision. The method achieves state-of-the-art results on ScanNet v2 and competitive performance on ScanNet200, ScanNet++ and Matterport3D, with clear improvements on geometrically challenging classes. By leveraging Gaussian opacity and scale attributes, G2P enhances boundary localization and object discrimination, offering a practical, 3D-only approach to appearance-consistent semantic segmentation.

Abstract

Semantic segmentation on point clouds is critical for 3D scene understanding. However, sparse and irregular point distributions provide limited appearance evidence, making geometry-only features insufficient to distinguish objects with similar shapes but distinct appearances (e.g., color, texture, material). We propose Gaussian-to-Point (G2P), which transfers appearance-aware attributes from 3D Gaussian Splatting to point clouds for more discriminative and appearance-consistent segmentation. Our G2P address the misalignment between optimized Gaussians and original point geometry by establishing point-wise correspondences. By leveraging Gaussian opacity attributes, we resolve the geometric ambiguity that limits existing models. Additionally, Gaussian scale attributes enable precise boundary localization in complex 3D scenes. Extensive experiments demonstrate that our approach achieves superior performance on standard benchmarks and shows significant improvements on geometrically challenging classes, all without any 2D or language supervision.

Figures (4)

• Figure 1: G2P augments Gaussian attributes to points while preserving geometry. (a) Point Cloud. Traditional point cloud representations lack sufficient appearance information to distinguish objects with similar geometry. As shown in the prediction, the existing model fails to segment coplanar windows cleanly and misclassifies the metallic refrigerator (yellow boxes), revealing geometric bias from insufficient appearance cues. (b) GS-Augmented Point Cloud (ours). Through our novel Gaussian-to-Point feature augmentation, G2P enriches points with Gaussian attributes $(\Sigma, \alpha)$, which encode rich appearance and geometric information. This enables successful segmentation of both challenging cases, resolving the geometric bias problem.
• Figure 2: Overview of the Gaussian-to-Point (G2P) learning approach. G2P comprises a preparation stage and a main training stage. In the preparation stage, (a) Gaussian-to-Point Feature Augmentation aligns 3D Gaussian ($\mu^g, \Sigma, \alpha$) and point ($\mu^p, c, n$), aggregating Gaussian attributes ($\alpha, S$), where $S$ represents the scale derived from the $\Sigma$, into augmented point features ($\alpha', S'$). (b) Scale-based Boundary Extraction uses the aggregated Gaussian scale magnitude to derive boundary pseudo-labels. An appearance encoder is pre-trained in a self-supervised manner using the augmented point representation ($\mu^p, c, \alpha'$). In the training stage, (c) GS Appearance Distillation transfers appearance cues from the pre-trained encoder to the 3D segmentation model, which takes point cloud inputs ($\mu^p, c, n$). The model is trained under joint supervision from semantic labels, boundary pseudo-labels, and appearance distillation.
• Figure 3: Gaussian Scale distribution. (a) Points after alignment, color-coded by aggregated Gaussian scale magnitude (magenta = small, yellow = large). Small scales concentrate at object boundaries. (b) Extracted boundary pseudo-labels by filtering out large-scale points.
• Figure 4: Qualitative comparison on geometrically challenging classes in ScanNet v2.Yellow boxes highlight regions with differing results between models. Boxes with stars ($\star$) indicate category confusion, where baselines misclassify objects (shown in blue) due to geometric bias (e.g., door $\rightarrow$ wall, refrigerator $\rightarrow$ cabinet). Boxes without stars indicate incomplete segmentation, where baselines fail to cover the full extent of thin or coplanar structures. G2P mitigates both failure modes through opacity-guided appearance distillation and scale-based boundary extraction. Corresponding regions in Ground Truth (bottom row) are marked for reference. Best viewed in color with zoom.