Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Exploration

TL;DR

This work tackles the challenge of sequential embodied reasoning, where agents must reuse spatial knowledge across subtasks and identify infeasible goals. It introduces SEER-Bench to evaluate sequential EQA and EMN with both feasible and infeasible tasks, and 3DSPMR, a 3D spatial memory system that fuses relational, visual, and FoV-based geometric cues to enhance MLLM-driven reasoning and exploration. Through a unified spatial memory and an agentic reasoning framework with a geometric examination mechanism, 3DSPMR improves both correctness and exploration efficiency, establishing state-of-the-art performance on sequential tasks. The approach reduces memory overhead via novelty-driven FoV keyframes and demonstrates robust results across backbones, with strong implications for real-world long-horizon embodied AI.

Abstract

Existing research on indoor embodied tasks typically requires agents to actively explore unknown environments and reason about the scene to achieve a specific goal. However, when deployed in real life, agents often face sequential tasks, where each new sub-task follows the completion of the previous one, and certain sub-tasks may be infeasible, such as searching for a non-existent object. Compared with the single-task setting, the core challenge lies in reusing spatial knowledge accumulated from previous explorations to support subsequent reasoning and exploration. In this work, we investigate this underexplored yet practically significant embodied AI challenge. To evaluate this challenge, we introduce SEER-Bench, a new Sequential Embodied Exploration and Reasoning Benchmark encompassing encompassing two classic embodied tasks: Embodied Question Answering (EQA) and Embodied Multi-modal Navigation (EMN). Building on SEER-Bench, we propose 3DSPMR, a 3D SPatial Memory Reasoning approach that exploits relational, visual, and geometric cues from explored regions to augment Multi-Modal Large Language Models (MLLMs) for reasoning and exploration in sequential embodied tasks. To the best of our knowledge, this is the first work to explicitly incorporate geometric information into MLLM-based spatial understanding and reasoning. Extensive experiments verify that 3DSPMR achieves substantial performance gains on both sequential EQA and EMN tasks.

Figures (15)

• Figure 1: Illustration of 3DSPMR completing a sequential EQA task. After navigating to the kitchen and answering the first question, the agent receives a second query. Using its spatial memory, it recognizes two previously passed but insufficiently explored bathrooms, revisits them, and answers the question after sufficient exploration. For the third question, the agent locates the study but cannot find a bookshelf; once the study is fully explored and no alternative candidate rooms remain, 3DSPMR correctly identifies the task as infeasible.
• Figure 2: Unanswerable Question Examples and data statistics of EQA in SEER-Bench.
• Figure 3: The Pipeline of 3DSPMR. Our approach maintains a unified spatial memory fusing global relational, geometric, and local visual cues. This grounds the Agentic Reasoning Module to verify task feasibility and decide whether to answer or explore, while the Geo-Sem Exploration Module guides the agent toward semantically relevant frontiers when further information is required.
• Figure 4: Illustration for the Agentic Embodied Reasoning framework in 3DSPMR for the EQA task. Fig. (a) shows how MLLM-based modules leverage the 3D scene graph and stored visual cues in spatial memory for reasoning, while Fig. (b) illustrates how the geometric examination mechanism utilizes FoV coverage information to determine task completion.
• Figure 5: Data statistics of the EMN track in SEER-Bench and infeasible goal examples.