Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GSMem: 3D Gaussian Splatting as Persistent Spatial Memory for Zero-Shot Embodied Exploration and Reasoning

Yiren Lu, Yi Du, Disheng Liu, Yunlai Zhou, Chen Wang, Yu Yin

Abstract

Effective embodied exploration requires agents to accumulate and retain spatial knowledge over time. However, existing scene representations, such as discrete scene graphs or static view-based snapshots, lack \textit{post-hoc re-observability}. If an initial observation misses a target, the resulting memory omission is often irrecoverable. To bridge this gap, we propose \textbf{GSMem}, a zero-shot embodied exploration and reasoning framework built upon 3D Gaussian Splatting (3DGS). By explicitly parameterizing continuous geometry and dense appearance, 3DGS serves as a persistent spatial memory that endows the agent with \textit{Spatial Recollection}: the ability to render photorealistic novel views from optimal, previously unoccupied viewpoints. To operationalize this, GSMem employs a retrieval mechanism that simultaneously leverages parallel object-level scene graphs and semantic-level language fields. This complementary design robustly localizes target regions, enabling the agent to ``hallucinate'' optimal views for high-fidelity Vision-Language Model (VLM) reasoning. Furthermore, we introduce a hybrid exploration strategy that combines VLM-driven semantic scoring with a 3DGS-based coverage objective, balancing task-aware exploration with geometric coverage. Extensive experiments on embodied question answering and lifelong navigation demonstrate the robustness and effectiveness of our framework

GSMem: 3D Gaussian Splatting as Persistent Spatial Memory for Zero-Shot Embodied Exploration and Reasoning

Abstract

Effective embodied exploration requires agents to accumulate and retain spatial knowledge over time. However, existing scene representations, such as discrete scene graphs or static view-based snapshots, lack \textit{post-hoc re-observability}. If an initial observation misses a target, the resulting memory omission is often irrecoverable. To bridge this gap, we propose \textbf{GSMem}, a zero-shot embodied exploration and reasoning framework built upon 3D Gaussian Splatting (3DGS). By explicitly parameterizing continuous geometry and dense appearance, 3DGS serves as a persistent spatial memory that endows the agent with \textit{Spatial Recollection}: the ability to render photorealistic novel views from optimal, previously unoccupied viewpoints. To operationalize this, GSMem employs a retrieval mechanism that simultaneously leverages parallel object-level scene graphs and semantic-level language fields. This complementary design robustly localizes target regions, enabling the agent to ``hallucinate'' optimal views for high-fidelity Vision-Language Model (VLM) reasoning. Furthermore, we introduce a hybrid exploration strategy that combines VLM-driven semantic scoring with a 3DGS-based coverage objective, balancing task-aware exploration with geometric coverage. Extensive experiments on embodied question answering and lifelong navigation demonstrate the robustness and effectiveness of our framework
Paper Structure (14 sections, 12 equations, 5 figures, 4 tables)

This paper contains 14 sections, 12 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. 3DGS Mapping
  5. Multi-level Retrieval-Rendering
  6. ROI rendering & Vision-Language Reasoning
  7. Hybrid Exploration Strategy
  8. Experiments
  9. Active Embodied Question Answering
  10. Multi-modal Lifelong Navigation
  11. Case Analysis
  12. Ablation Study
  13. Real-world Practicality
  14. Conclusion

Figures (5)

  • Figure 1: With GS-Mem, previously explored regions can be retrieved and re-observed directly from the 3DGS memory without physically navigating to them.
  • Figure 2: Demonstration of Multi-level Retrieval-Rendering. Given a task-related target, the agent retrieves ROIs based on object-level and semantic-level cues. Subsequent viewpoint selection and rendering enable the agent to re-observe these regions for further reasoning.
  • Figure 3: Demonstration of our Hybrid Exploration Strategy. When frontier observations do not contain sufficient task-related cues for the VLM to make a decision, we incorporate an information gain-based score to select the most informative frontier for further exploration.
  • Figure 4: Case analysis. We analyze several cases where scene-graph and view-based representations fail, and demonstrate the advantages of 3DGS-based memory. The images shown correspond to the views selected by the VLM for answering the questions. The examples (a-c) illustrate failures of the scene-graph detector, while the last example (d) highlights how optimal viewpoint rendering benefits the VLM's reasoning.
  • Figure 5: Runtime analysis for real-world multi-process setup.