Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views

TL;DR

The paper tackles the challenge of spatial reasoning from limited views by enabling vision-language models to imagine 3D scenes during reasoning. It introduces 3DThinker, which injects compact 3D latent tokens into the reasoning process and aligns them with a 3D foundation model (VGGT) via a two-stage training regime: supervised 3D latent alignment and outcome-driven reinforcement learning. The approach does not rely on densely labeled 3D data or external priors during inference, yet can recover 3D representations to support interpretability. Empirical results show consistent improvements across multiple benchmarks and base models, with strong generalization to unseen datasets and tasks.

Abstract

Though recent advances in vision-language models (VLMs) have achieved remarkable progress across a wide range of multimodal tasks, understanding 3D spatial relationships from limited views remains a significant challenge. Previous reasoning methods typically rely on pure text (e.g., topological cognitive maps) or on 2D visual cues. However, their limited representational capacity hinders performance in specific tasks that require 3D spatial imagination. To address this limitation, we propose 3DThinker, a framework that can effectively exploits the rich geometric information embedded within images while reasoning, like humans do. Our framework is the first to enable 3D mentaling during reasoning without any 3D prior input, and it does not rely on explicitly labeled 3D data for training. Specifically, our training consists of two stages. First, we perform supervised training to align the 3D latent generated by VLM while reasoning with that of a 3D foundation model (e.g., VGGT). Then, we optimize the entire reasoning trajectory solely based on outcome signals, thereby refining the underlying 3D mentaling. Extensive experiments across multiple benchmarks show that 3DThinker consistently outperforms strong baselines and offers a new perspective toward unifying 3D representations into multimodal reasoning. Our code will be available at https://github.com/zhangquanchen/3DThinker.

Figures (4)

• Figure 1: Illustration of our 3DThinker. Existing methods typically perform reasoning based solely on pure text or 2D visual cues, without fully exploiting the rich spatial and geometric information inherent in images. Other methods attempt to enhance the input by introducing auxiliary modalities (e.g., depth maps or coordinates), yet these often depend on additional annotations or external tools. In contrast, our framework enables VLMs to intrinsically form 3D mental representations during reasoning, thereby improving their spatial understanding.
• Figure 2: The schematic illustration of our 3DThinker, a framework that enables thinking with 3D mentaling. (1) Stage 1: 3DThinker is first trained under supervision using our constructed CoT data (see Sec. \ref{['sec:data']}), aligning the generated 3D latents with the feature space of 3D foundation model. This alignment allows the model to leverage suitable 3D spatial mentaling while reasoning. (2) Stage 2: After supervised training, we further optimize the entire trajectory using only outcome signals, while maintaining the alignment of the 3D latents.
• Figure 3: Illustration of our projector, which transforms VLM-generated 3D latent into the feature space of VGGT.
• Figure 4: The reasoning process for different cases is presented, along with the visualization of the 3D latent representations.