Unleashing Spatial Reasoning in Multimodal Large Language Models via Textual Representation Guided Reasoning

Abstract

Existing Multimodal Large Language Models (MLLMs) struggle with 3D spatial reasoning, as they fail to construct structured abstractions of the 3D environment depicted in video inputs. To bridge this gap, drawing inspiration from cognitive theories of allocentric spatial reasoning, we investigate how to enable MLLMs to model and reason over text-based spatial representations of video. Specifically, we introduce Textual Representation of Allocentric Context from Egocentric Video (TRACE), a prompting method that induces MLLMs to generate text-based representations of 3D environments as intermediate reasoning traces for more accurate spatial question answering. TRACE encodes meta-context, camera trajectories, and detailed object entities to support structured spatial reasoning over egocentric videos. Extensive experiments on VSI-Bench and OST-Bench demonstrate that TRACE yields notable and consistent improvements over prior prompting strategies across a diverse range of MLLM backbones, spanning different parameter scales and training schemas. We further present ablation studies to validate our design choices, along with detailed analyses that probe the bottlenecks of 3D spatial reasoning in MLLMs.

Figures (6)

• Figure 1: Motivation for Textual Representation of Allocentric Context from Egocentric Video (TRACE) in video-based spatial reasoning. (a) An egocentric video paired with a query that requires holistic spatial reasoning. (b) A textual description that vividly captures the room layout needed to solve the example spatial question answering (QA). (c) TRACE encodes meta-context, camera trajectory, and entities, serving as an intermediate reasoning trace for spatial QA with MLLMs.
• Figure 2: Illustration of our Textual Representation of Allocentric Context from Egocentric Video (TRACE). We construct TRACE by aligning a global coordinate system with the room layout and geometry, logging the camera trajectory across temporal steps, and registering visible objects with key attributes, estimated positions, and spatial relations. Here, we also show the key prompts used to guide MLLMs to generate this intermediate reasoning trace.
• Figure 3: Performance gains across models on VSI-Bench. TRACE yields consistent, state-of-the-art performance gains compared to Direct prompting baselines, across various model architectures and parameter scales.
• Figure 4: Decompositional analysis of the reasoning parser and spatial descriptor. The Qwen series lags behind the state-of-the-art Gemini 3 on both spatial reasoning and visual perception.
• Figure 5: A visual illustration demonstrates that TRACE is more effective than the cognitive map (CM) approach. Notably, the CM lacks the 3D granularity required for many spatial reasoning tasks.