Map2Thought: Explicit 3D Spatial Reasoning via Metric Cognitive Maps

TL;DR

Map2Thought tackles the lack of explicit spatial grounding in 3D Vision-Language Models by introducing a dual-format Metric-CogMap and an explicit Cog-CoT reasoning module. The framework combines a discrete grid with continuous 3D geometry to support both symbolic relational reasoning and precise metric measurements, enabling verifiable geometric traces. Empirical results on VSI-Bench demonstrate enhanced data efficiency and competitive accuracy under reduced supervision, outperforming several baselines. This work advances interpretable, robust 3D spatial understanding in multimodal systems and provides a scalable pipeline for explicit spatial reasoning.

Abstract

We propose Map2Thought, a framework that enables explicit and interpretable spatial reasoning for 3D VLMs. The framework is grounded in two key components: Metric Cognitive Map (Metric-CogMap) and Cognitive Chain-of-Thought (Cog-CoT). Metric-CogMap provides a unified spatial representation by integrating a discrete grid for relational reasoning with a continuous, metric-scale representation for precise geometric understanding. Building upon the Metric-CogMap, Cog-CoT performs explicit geometric reasoning through deterministic operations, including vector operations, bounding-box distances, and occlusion-aware appearance order cues, producing interpretable inference traces grounded in 3D structure. Experimental results show that Map2Thought enables explainable 3D understanding, achieving 59.9% accuracy using only half the supervision, closely matching the 60.9% baseline trained with the full dataset. It consistently outperforms state-of-the-art methods by 5.3%, 4.8%, and 4.0% under 10%, 25%, and 50% training subsets, respectively, on the VSI-Bench.

Figures (11)

• Figure 1: Comparison with prior approaches and efficiency gains. (a) Previous 3D VLMs fuse visual and geometric tokens but through implicit latent reasoning, limiting spatial interpretability. (b) Our approach introduces a metrically grounded cognitive map (Metric-CogMap) and a chain-of-thought-style reasoning process (Cog-CoT), enabling explicit and interpretable spatial reasoning from the same multimodal inputs. (c) This design yields substantial data efficiency: with just 10% or 25% of the training data, our model is comparable to or surpasses the performance of the baseline model trained with 50% of the data.
• Figure 2: Overview of our method. Given an RGB video and a language query, we extract 2D image tokens and 3D geometry-aware tokens, fuse them into 3D-aware visual tokens, and input them to the VLM. The Metric-CogMap (orange blocks) encodes the scene using both a discrete grid and a metric-scale spatial representation. Cog-CoT (right, grey panel) then performs explicit and deterministic geometric reasoning over the map, yielding a transparent and interpretable answer.
• Figure 3: Visualization of a Metric-CogMap example. Given the reference frame shown at the top left, the discrete grid representation (top two) assigns each object an integer-valued position, while the box occupancy map encodes the grid ranges each object occupies. The metric-scale representation (bottom two) records object centroids and axis-aligned bounding boxes (AABBs) in a globally aligned real-world coordinate system.
• Figure 4: Overview of the Metric-CogMap construction pipeline. Given video and text inputs, the pipeline selects 64 uniformly distributed frames while ensuring key frames containing the highest-confidence detections of text-referenced objects (green outlines and annotations). For categories requiring multi-instance detection (e.g., the refrigerator), covisibility maps anchor the highest-confidence frame and suppress redundant detections by checking whether candidate regions appeared in earlier selected views. Only geometrically unique detections within valid mask regions are retained. These detections are propagated across frames, and multi-view segments are merged with consistent IDs during final integration. The example on the right shows the resulting detections for several question-relevant categories (e.g., sofa, trash bin, chair, table) from scene0651_02 in the ScanNet dataset dai2017scannet, an evaluation scene in VSI-Bench vsibench.
• Figure 5: Cog-CoT workflow. Given a query, the Metric-CogMap, and the corresponding Task-Specific instruction, the system retrieves the scene-level Metric-CogMap, and Cog-CoT executes the geometric computations step-by-step, yielding verifiable intermediate numerical evidence used to generate the final answer.