ProFuse: Efficient Cross-View Context Fusion for Open-Vocabulary 3D Gaussian Splatting

TL;DR

ProFuse addresses open-vocabulary 3D scene understanding by grounding 3D Gaussian Splatting in dense multi-view correspondences. It introduces a dense correspondence pre-registration phase to initialize geometry and form 3D Context Proposals, then attaches global semantics to Gaussians via visibility-weighted fusion without any render-time language supervision. A lightweight inference pipeline using cosine similarity and Product Quantization enables fast 3D retrieval and semantic activation of Gaussians. The approach achieves strong open-vocabulary object selection and point-cloud understanding while reducing processing to about five minutes per scene, roughly 2× faster than previous state-of-the-art, by avoiding heavy densification and gradient-based semantic training. Overall, ProFuse provides an efficient, registration-based route to coherent open-vocabulary 3D semantics grounded in cross-view correspondences.

Abstract

We present ProFuse, an efficient context-aware framework for open-vocabulary 3D scene understanding with 3D Gaussian Splatting (3DGS). The pipeline enhances cross-view consistency and intra-mask cohesion within a direct registration setup, adding minimal overhead and requiring no render-supervised fine-tuning. Instead of relying on a pretrained 3DGS scene, we introduce a dense correspondence-guided pre-registration phase that initializes Gaussians with accurate geometry while jointly constructing 3D Context Proposals via cross-view clustering. Each proposal carries a global feature obtained through weighted aggregation of member embeddings, and this feature is fused onto Gaussians during direct registration to maintain per-primitive language coherence across views. With associations established in advance, semantic fusion requires no additional optimization beyond standard reconstruction, and the model retains geometric refinement without densification. ProFuse achieves strong open-vocabulary 3DGS understanding while completing semantic attachment in about five minutes per scene, which is two times faster than SOTA.

Figures (5)

• Figure 1: Overview of ProFuse. Left: A dense matcher supplies cross-view geometric and semantic correspondences. Top: Warped masks are grouped into 3D Context Proposals with a shared global feature. Bottom: Triangulated matches initialize a compact Gaussian scene, and proposal features are fused without render supervision for coherent open-vocabulary 3D semantics.
• Figure 2: Pre-registration. For each reference view we select $K$ neighbors via view clustering, then apply a pre-trained dense matcher to obtain per-pixel warps $W_{j\!\to i}$ and confidences $\alpha_{j\!\to i}$. Bottom right: Given the warps of a pixel pair, we triangulate a 3D seed point for Gaussian initialization. Top right: Warped IoU comparison on every reference–neighbor mask pair; masks that pass the selection form edges of a bipartite graph.
• Figure 3: From context proposal to global feature. Left: masks of the same entity are grouped into a 3D Context Proposal. Center: for a pixel $p$, the renderer returns the top-$K$ Gaussians with contributions $\{\omega_{i,p,t}\}_{t=1}^{K}$, from which the mask mass$\mu\!\left(M_i^k\right)$ is computed. Right: a mass-weighted pool of member mask embeddings forms the proposal feature, which is registered to Gaussians via Eq. (\ref{['eq:accumulate']}).
• Figure 4: Qualitative comparison of object-level semantic queries on the LERF-OVS lerf2023 dataset. Our method produces more accurate and cleaner object retrieval, showing sharper correspondence between the text query and the selected 3D content.
• Figure 5: Feature visualizations on the ScanNet dai2017scannetrichlyannotated3dreconstructions dataset using registration-based methods. Colors represent normalized language features transferred to mesh vertices and rendered via a fixed RGB projection. ProFuse produces cleaner regions with sharper boundaries and fewer speckles.