3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning

TL;DR

3D-Mem introduces Memory Snapshots and Frontier Snapshots as a scalable, image-based 3D scene memory for embodied agents, enabling lifelong exploration and reasoning through incremental construction and Prefiltering. By integrating frontier-based exploration and leveraging Vision-Language Models to reason over multi-view memory, it outperforms object-centric and dense-geometry baselines on A-EQA, EM-EQA, and GOAT-Bench. The approach demonstrates superior spatial reasoning, memory efficiency, and adaptability to lifelong tasks, with comprehensive ablations and analyses illustrating component contributions and trade-offs. This framework advances practical embodied AI by combining compact visual memory with active exploration strategies and VLM-driven reasoning.

Abstract

Constructing compact and informative 3D scene representations is essential for effective embodied exploration and reasoning, especially in complex environments over extended periods. Existing representations, such as object-centric 3D scene graphs, oversimplify spatial relationships by modeling scenes as isolated objects with restrictive textual relationships, making it difficult to address queries requiring nuanced spatial understanding. Moreover, these representations lack natural mechanisms for active exploration and memory management, hindering their application to lifelong autonomy. In this work, we propose 3D-Mem, a novel 3D scene memory framework for embodied agents. 3D-Mem employs informative multi-view images, termed Memory Snapshots, to represent the scene and capture rich visual information of explored regions. It further integrates frontier-based exploration by introducing Frontier Snapshots-glimpses of unexplored areas-enabling agents to make informed decisions by considering both known and potential new information. To support lifelong memory in active exploration settings, we present an incremental construction pipeline for 3D-Mem, as well as a memory retrieval technique for memory management. Experimental results on three benchmarks demonstrate that 3D-Mem significantly enhances agents' exploration and reasoning capabilities in 3D environments, highlighting its potential for advancing applications in embodied AI.

Figures (18)

• Figure 1: With 3D-Mem, explored regions are represented by a set of Memory Snapshots capturing clusters of co-visible objects, i.e., the objects observable in a single image observation, along with their spatial relationships and background context, as shown in the bottom-left example. Unexplored regions are represented by navigable frontiers along with image observations, referred to as Frontier Snapshots.
• Figure 2: The memory aggregation process of 3D-Mem. At each step $t$, the object set $\mathcal{O}_t$ is first updated using the object-wise update pipeline introduced in Section \ref{['approach_mem_construction']}. The newly detected objects and the updated existing objects are then jointly clustered into new memory snapshots using co-visibility clustering (Algorithm \ref{['pseudo:static']}), which are used to update the memory snapshot set $\mathcal{S}_t$.
• Figure 3: 3D-Mem as visual input for the VLM in embodied question answering. The VLM first retrieves relevant memory snapshots with prefiltering, then utilizes the frontier snapshots and memory snapshots to perceive the scene and reason about the embodied questions.
• Figure 4: Frame Efficiency of 3D-Mem on EM-EQA. LLM-Match Score vs. Average Number of Frames for 3D-Mem and Multi-Frame both using GPT-4o
• Figure 5: Failure Case 1: Some questions in A-EQA are vague and may have multiple reasonable answers.