Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RieMind: Geometry-Grounded Spatial Agent for Scene Understanding

Fernando Ropero, Erkin Turkoz, Daniel Matos, Junqing Du, Antonio Ruiz, Yanfeng Zhang, Lu Liu, Mingwei Sun, Yongliang Wang

Abstract

Visual Language Models (VLMs) have increasingly become the main paradigm for understanding indoor scenes, but they still struggle with metric and spatial reasoning. Current approaches rely on end-to-end video understanding or large-scale spatial question answering fine-tuning, inherently coupling perception and reasoning. In this paper, we investigate whether decoupling perception and reasoning leads to improved spatial reasoning. We propose an agentic framework for static 3D indoor scene reasoning that grounds an LLM in an explicit 3D scene graph (3DSG). Rather than ingesting videos directly, each scene is represented as a persistent 3DSG constructed by a dedicated perception module. To isolate reasoning performance, we instantiate the 3DSG from ground-truth annotations. The agent interacts with the scene exclusively through structured geometric tools that expose fundamental properties such as object dimensions, distances, poses, and spatial relationships. The results we obtain on the static split of VSI-Bench provide an upper bound under ideal perceptual conditions on the spatial reasoning performance, and we find that it is significantly higher than previous works, by up to 16\%, without task specific fine-tuning. Compared to base VLMs, our agentic variant achieves significantly better performance, with average improvements between 33\% to 50\%. These findings indicate that explicit geometric grounding substantially improves spatial reasoning performance, and suggest that structured representations offer a compelling alternative to purely end-to-end visual reasoning.

RieMind: Geometry-Grounded Spatial Agent for Scene Understanding

Abstract

Visual Language Models (VLMs) have increasingly become the main paradigm for understanding indoor scenes, but they still struggle with metric and spatial reasoning. Current approaches rely on end-to-end video understanding or large-scale spatial question answering fine-tuning, inherently coupling perception and reasoning. In this paper, we investigate whether decoupling perception and reasoning leads to improved spatial reasoning. We propose an agentic framework for static 3D indoor scene reasoning that grounds an LLM in an explicit 3D scene graph (3DSG). Rather than ingesting videos directly, each scene is represented as a persistent 3DSG constructed by a dedicated perception module. To isolate reasoning performance, we instantiate the 3DSG from ground-truth annotations. The agent interacts with the scene exclusively through structured geometric tools that expose fundamental properties such as object dimensions, distances, poses, and spatial relationships. The results we obtain on the static split of VSI-Bench provide an upper bound under ideal perceptual conditions on the spatial reasoning performance, and we find that it is significantly higher than previous works, by up to 16\%, without task specific fine-tuning. Compared to base VLMs, our agentic variant achieves significantly better performance, with average improvements between 33\% to 50\%. These findings indicate that explicit geometric grounding substantially improves spatial reasoning performance, and suggest that structured representations offer a compelling alternative to purely end-to-end visual reasoning.
Paper Structure (14 sections, 1 equation, 3 figures, 4 tables)

This paper contains 14 sections, 1 equation, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. VLMs for Indoor Spatial QA.
  4. Tool Integrated Spatial Reasoning.
  5. 3DSG for Spatial Q&A.
  6. Geometry-Grounded Agentic Framework
  7. Perception Layer
  8. Reasoning Layer
  9. Agentic Architecture and Prompt.
  10. Geometry-Grounded Toolbox.
  11. Experiments
  12. Impact of Agentic Reasoning
  13. Upper Bound Evaluation on VSI-Bench
  14. Conclusion

Figures (3)

  • Figure 1: RieMind: Geometry-grounded framework for spatial agentic scene understanding, and in this figure, it is reasoning over the relative direction of objects in a scene, while performing allocentric-egocentric frame transformations.
  • Figure 2: Reasoning component of the spatial agent framework, containing the different types of tools, and general agent architecture following the MCP.
  • Figure 3: Exemplifying the different types of explicit grounding through a relative direction question. For any question posed to the agent, it will need to disambiguate the actual entity, and this grounds its reasoning. For this question, it then needs to obtain positions, orientations, and create the specific egocentric frame with its own axis convention. It can then perform the projection of the desired object into this new egocentric frame. All these tools and processes ground the reasoning sequence of the agent, and force it to stay focused on these basic geometric objects.