CaRF: Enhancing Multi-View Consistency in Referring 3D Gaussian Splatting Segmentation

TL;DR

CaRF tackles multi-view inconsistency in Referring 3D Gaussian Splatting Segmentation by learning directly in 3D Gaussian space with camera-aware conditioning and paired-view supervision. The key ideas are Gaussian Field Camera Encoding (GFCE), which injects camera geometry into Gaussian–text interactions, and In-Training Paired-View Supervision (ITPVS), which enforces cross-view consistency by jointly supervising the same Gaussians across two calibrated views. Together, these components yield robust view-consistent language grounding and improved segmentation accuracy, achieving up to +16.8% mIoU on Ref-LERF, +4.3% on LERF-OVS, and +2.0% on 3D-OVS over prior methods. The approach advances 3D open-vocabulary segmentation and has practical implications for embodied AI, AR/VR, and autonomous perception by providing more reliable cross-view reasoning in Gaussian-based scene representations.

Abstract

Referring 3D Gaussian Splatting Segmentation (R3DGS) aims to interpret free-form language expressions and localize the corresponding 3D regions in Gaussian fields. While recent advances have introduced cross-modal alignment between language and 3D geometry, existing pipelines still struggle with cross-view consistency due to their reliance on 2D rendered pseudo supervision and view specific feature learning. In this work, we present Camera Aware Referring Field (CaRF), a fully differentiable framework that operates directly in the 3D Gaussian space and achieves multi view consistency. Specifically, CaRF introduces Gaussian Field Camera Encoding (GFCE), which incorporates camera geometry into Gaussian text interactions to explicitly model view dependent variations and enhance geometric reasoning. Building on this, In Training Paired View Supervision (ITPVS) is proposed to align per Gaussian logits across calibrated views during training, effectively mitigating single view overfitting and exposing inter view discrepancies for optimization. Extensive experiments on three representative benchmarks demonstrate that CaRF achieves average improvements of 16.8%, 4.3%, and 2.0% in mIoU over state of the art methods on the Ref LERF, LERF OVS, and 3D OVS datasets, respectively. Moreover, this work promotes more reliable and view consistent 3D scene understanding, with potential benefits for embodied AI, AR/VR interaction, and autonomous perception.

Figures (6)

• Figure 1: Qualitative comparison between ReferSplatReferSplat and our CaRF on the Ramen and Waldo Kitchen scenes. ReferSplat fails to maintain multi-view consistency: in Ramen, the glass of water is completely missed in View 1 and exhibits severe rendering artifacts in View 2; in Waldo Kitchen, the "refrigerator" mask leaks into adjacent regions and shows fragmented boundaries. In contrast, CaRF produces coherent and geometrically consistent masks across views, effectively preserving details of fine-grained objects.
• Figure 2: Motivation of our method. (a) The previous method performs per-view rasterization and supervision with a single GT mask at an iteration, which can introduce inconsistencies across views. (b) CaRF adds multi-view joint supervision during training to explicitly enforce cross-view agreement, resulting in consistent, artifact-resistant masks across views at inference.
• Figure 3: Overview of the proposed CaRF framework. Given calibrated multi-view RGB images and a text query, CaRF first generates robust pseudo masks through confidence-weighted selection. Each Gaussian is augmented with a referring feature that interacts with language embeddings via a cross-interaction module. A camera-aware encoding then integrates geometric information from camera intrinsics and extrinsics to enhance view consistency. Finally, referring features are rasterized into 2D response maps and optimized under in-training supervision to achieve geometry-aware, view-consistent referring segmentation.
• Figure 4: Qualitative comparisons on the Ref-LERF dataset across two representative scenes and two calibrated views. Each row shows results from GS-Groupingye2024gaussian, ReferSplatReferSplat, our CaRF, and the GT masks. GS-Grouping often confuses nearby objects with similar appearance due to the lack of fine-grained spatial reasoning, while ReferSplat produces view-inconsistent or incomplete segmentations because supervision is derived from single-view pseudo masks. In contrast, CaRF generates spatially coherent and semantically precise masks across multiple views by leveraging camera-aware geometry and paired-view consistency constraints.
• Figure 5: Failure cases on the Ref-LERF dataset across two representative scenes and two calibrated views. Each row shows results from ReferSplatReferSplat, our CaRF, and the GT masks. Most errors arise from inaccurate or misaligned pseudo GT masks (see the last row).