Think3D: Thinking with Space for Spatial Reasoning

TL;DR

Think3D presents a training-free framework for spatial reasoning in vision–language models by attaching a 3D reconstruction–based manipulation toolkit and an iterative observe–manipulate–reflect loop that anchors reasoning to camera poses. By enabling explicit 3D interaction with reconstructed point clouds and rendering novel views in global or ego-centric modes, Think3D yields strong cross-benchmark gains on BLINK Multi-view, MindCube, and VSI-Bench for large models, and, with Think3D-RL, substantial improvements for smaller models. The RL component learns informative exploration policies that significantly boost the utility of 3D tool use, increasing gains from baseline levels (e.g., +0.7% to +6.8% on some setups). Overall, Think3D demonstrates that training-free, tool-augmented spatial exploration is a viable path toward more flexible and human-like 3D reasoning in multimodal agents, with broad implications for spatial intelligence in AI systems.

Abstract

Understanding and reasoning about the physical world requires spatial intelligence: the ability to interpret geometry, perspective, and spatial relations beyond 2D perception. While recent vision large models (VLMs) excel at visual understanding, they remain fundamentally 2D perceivers and struggle with genuine 3D reasoning. We introduce Think3D, a framework that enables VLM agents to think with 3D space. By leveraging 3D reconstruction models that recover point clouds and camera poses from images or videos, Think3D allows the agent to actively manipulate space through camera-based operations and ego/global-view switching, transforming spatial reasoning into an interactive 3D chain-of-thought process. Without additional training, Think3D significantly improves the spatial reasoning performance of advanced models such as GPT-4.1 and Gemini 2.5 Pro, yielding average gains of +7.8% on BLINK Multi-view and MindCube, and +4.7% on VSI-Bench. We further show that smaller models, which struggle with spatial exploration, benefit significantly from a reinforcement learning policy that enables the model to select informative viewpoints and operations. With RL, the benefit from tool usage increases from +0.7% to +6.8%. Our findings demonstrate that training-free, tool-augmented spatial exploration is a viable path toward more flexible and human-like 3D reasoning in multimodal agents, establishing a new dimension of multimodal intelligence. Code and weights are released at https://github.com/zhangzaibin/spagent.

Figures (22)

• Figure 1: Comparison between prior "think with image" zheng2025deepeyes and our proposed "think with space". While the former reasons over 2D content by manipulating images, our method operates directly within 3D point cloud space for spatial understanding.
• Figure 2: The Think3D pipeline. The VLM interacts with the 3D scene through iterative calls to the 3D Manipulation Toolkit, issuing viewpoint-manipulation actions that control camera pose and rendering parameters. Each rendered image is appended to the agent’s memory and informs the next reasoning step, forming a repeated cycle of observe → manipulate → reflect.
• Figure 3: Spatial exploration behavior of Think3D. The agent autonomously selects viewpoints and switches between global and ego-centric views; after RL training, it explores angles more systematically than the untuned baseline.
• Figure 4: Task level spatial exploration patterns. This figure shows the distribution of viewpoint selections made by GPT-4.1 across different tasks. The exploration patterns vary substantially: for instance, tasks such as route planning exhibit a strong preference for top-down views (0,60), while others rely on more diverse or oblique perspectives.
• Figure 5: Reinforcement Learning Dynamics. As RL fine-tuning progresses, the model learns when extra 3D tool calls are worthwhile, shifting from shorter but less accurate trajectories to more informative explorations with higher reward.