GaussExplorer: 3D Gaussian Splatting for Embodied Exploration and Reasoning

TL;DR

GaussExplorer tackles embodied exploration and reasoning in 3D scenes by grounding natural-language queries directly in semantic 3D Gaussians via 3D Gaussian Splatting. It combines LLM-driven Gaussian search and clustering with VLM-guided novel-view synthesis to identify informative viewpoints and perform fine-grained 3D localization, including 3D referring segmentation. Key contributions include semantic Gaussian construction, a visibility-based initial view selection, a novel-view adjustment pipeline with VLMs as judges, and a verification step that enhances robustness. Empirical results on EM-EQA and a new 3D referring segmentation benchmark show state-of-the-art performance and demonstrate the practicality of integrating VLM reasoning with 3DGS for embodied tasks, enabling accurate reasoning and grounding in complex 3D environments.

Abstract

We present GaussExplorer, a framework for embodied exploration and reasoning built on 3D Gaussian Splatting (3DGS). While prior approaches to language-embedded 3DGS have made meaningful progress in aligning simple text queries with Gaussian embeddings, they are generally optimized for relatively simple queries and struggle to interpret more complex, compositional language queries. Alternative studies based on object-centric RGB-D structured memories provide spatial grounding but are constrained by pre-fixed viewpoints. To address these issues, GaussExplorer introduces Vision-Language Models (VLMs) on top of 3DGS to enable question-driven exploration and reasoning within 3D scenes. We first identify pre-captured images that are most correlated with the query question, and subsequently adjust them into novel viewpoints to more accurately capture visual information for better reasoning by VLMs. Experiments show that ours outperforms existing methods on several benchmarks, demonstrating the effectiveness of integrating VLM-based reasoning with 3DGS for embodied tasks.

Figures (15)

• Figure 1: GaussExplorer aims at the embodied exploration and reasoning based on 3D Gaussian Splatting. Given an input question, we first identify initial viewpoints by searching for relevant 3D Gaussians. These viewpoints are then refined into novel-view images with the VLM-as-Judge mechanism, which evaluates rendered views to maximize visual evidence and decide the final viewpoints. Finally, the final views are processed by a VLM to generate the response. We further extend our framework to support fine-grained 3D localization tasks with complex language queries such as 3D referring segmentation.
• Figure 2: Overview. (a) We first build a semantic 3DGS scene, where input views $\mathcal{I}$ and their semantic information produced by foundation models are lifted into 3D. (b) In initial view selection, (b-1) the query is first rephrased into relevant semantic categories $\mathcal{C}^{evidence}$ by LLM, (b-2) activated 3D Gaussians associated with those categories are grouped into spatial clusters, and (b-3) representative training camera poses covering these clusters are selected. (c) Finally, the selected views are rendered and passed to a Vision-Language Model (VLM) for fine-grained reasoning, which identifies the most informative views and produces the final answer to the query.
• Figure 3: Visibility score. We evaluate visibility by checking whether Gaussians with the highest rendering weight belong to the target instance cluster. If they align, the instance is considered visible; if occluded, the selected Gaussians originate from other regions, resulting in no overlap and a low visibility score.
• Figure 4: EQA pipeline comparison. The competing method 3d-mem attempts to answer queries directly using initial views selected from given images, lacking any viewpoint refinement. In contrast, our method automatically explores nearby novel viewpoints via VLM-based pose adjustment, maximizing visual evidence more clearly. Finally, a VLM-based verification step compares the initial and refined views to select the one providing the most reliable evidence. Note that $\mathcal{A}$ represents the predicted answer to the given question.
• Figure 5: Novel-view adjustment. From an initial camera pose, we generate multiple novel-view candidates and refine them through a novel view adjustment module. Each candidate's view is rendered and evaluated by a VLM with a visual-QA prompt to obtain initial answer predictions. These answers, together with the question, are then passed to an LLM to select the most informative final viewpoint.