Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps

Abstract

Precise spatial understanding from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs), as their visual representations are predominantly semantic and lack explicit geometric grounding. While existing approaches augment visual tokens with geometric cues from visual geometry models, their MLLM is still required to implicitly infer the underlying 3D structure of the scene from these augmented tokens, limiting its spatial reasoning capability. To address this issue, we introduce Cog3DMap, a framework that recurrently constructs an explicit 3D memory from multi-view images, where each token is grounded in 3D space and possesses both semantic and geometric information. By feeding these tokens into the MLLM, our framework enables direct reasoning over a spatially structured 3D map, achieving state-of-the-art performance on various spatial reasoning benchmarks. Code will be made publicly available.

Figures (6)

• Figure 1: Overall pipeline of Cog3DMap. (a) Given a sequence of multi-view images, our recurrent framework, Cog3DMap, progressively integrates visual observations into a unified 3D memory map. Each spatial coordinate in the map is associated with a token carrying both semantic and geometric information. (b) Then, the resulting compact and explicit 3D map is fed into the MLLM decoder for spatial reasoning.
• Figure 2: We compare accuracy (%) against the average number of visual tokens (log scale) for short, medium, and long horizon tasks. Cog3DMap(Ours) consistently dominates the trade-off, achieving comparable or better accuracy while requiring substantially fewer visual tokens (up to 90.2% reduction), demonstrating improved token efficiency across horizons.
• Figure 3: Visualization of attention scores over visual tokens across varying text queries. To analyze the model behavior, we fix the scene and vary the target object in the text query. Our Cog3DMap assigns high attention scores to visual tokens relevant to the query object, without explicit supervision for such attention.
• Figure 4: Visualization of attention scores over visual tokens on a validation sample from Scan2Cap chen2021scan2cap. Cog3DMap assigns high attention scores to the visual tokens corresponding to the generated answer.
• Figure 5: Visualization of attention scores over visual tokens on a validation sample from Scan2Cap chen2021scan2cap. Cog3DMap assigns high attention scores to the visual tokens corresponding to the generated answer.