Thinking with Blueprints: Assisting Vision-Language Models in Spatial Reasoning via Structured Object Representation

TL;DR

The paper tackles the difficulty of global spatial reasoning in vision–language models by introducing an object-centric blueprint that encodes object positions, sizes, and attributes in a JSON-like structure. It divides reasoning into observation (blueprint construction) and reflection (blueprint-based analysis) and trains the model in two stages: supervised fine-tuning on blueprint-embedded traces generated via a teacher VLM and Monte Carlo Tree Search, followed by reinforcement learning with blueprint-aware rewards (object cardinality and causal consistency) and anti-shortcut data augmentation. Empirical results across SAT, BLINK, RoboSpatial, and VSR benchmarks show state-of-the-art performance, with substantial gains in iid and out-of-distribution settings, and ablations confirm the necessity of the blueprint, rewards, and augmentation. The approach advances spatially grounded reasoning in VLMs and points to extensions for video and 3D reasoning with practical implications for robotics and grounded AI.

Abstract

Spatial reasoning -- the ability to perceive and reason about relationships in space -- advances vision-language models (VLMs) from visual perception toward spatial semantic understanding. Existing approaches either revisit local image patches, improving fine-grained perception but weakening global spatial awareness, or mark isolated coordinates, which capture object locations but overlook their overall organization. In this work, we integrate the cognitive concept of an object-centric blueprint into VLMs to enhance spatial reasoning. Given an image and a question, the model first constructs a JSON-style blueprint that records the positions, sizes, and attributes of relevant objects, and then reasons over this structured representation to produce the final answer. To achieve this, we introduce three key techniques: (1) blueprint-embedded reasoning traces for supervised fine-tuning to elicit basic reasoning skills; (2) blueprint-aware rewards in reinforcement learning to encourage the blueprint to include an appropriate number of objects and to align final answers with this causal reasoning; and (3) anti-shortcut data augmentation that applies targeted perturbations to images and questions, discouraging reliance on superficial visual or linguistic cues. Experiments show that our method consistently outperforms existing VLMs and specialized spatial reasoning models.

Figures (39)

• Figure 1: An illustrative comparison of our method with other spatial reasoning approaches. Inspired by the cognitive concept of an object-centric blueprint, our method first constructs a JSON-style blueprint recording the positions, sizes, and attributes of relevant objects, and then reasons over this structured representation to produce the final answer. Other approaches overlook such an explicit and global blueprint during reasoning, often resulting in superficial analysis and incorrect answers.
• Figure 2: Approach overview. (a): Construct blueprint-embedded reasoning traces. We prompt a strong teacher VLM to generate atomic reasoning steps, including adding objects to the blueprint, analyzing it, and producing the final answer. These steps are then assembled into coherent traces via MCTS. (b): Perform supervised fine-tuning. The model is fine-tuned on the blueprint-embedded reasoning traces to elicit basic reasoning skills. (c): Perform reinforcement learning. The overall reward composed of two conventional ones (answer correctness and trace format) alongside two blueprint-aware rewards: object cardinality reward, which encourages including an appropriate number of objects in the blueprint, and causal consistency reward, which ensures final answers are grounded in intermediate reasoning. Moreover, we employ anti-shortcut data augmentation, perturbing images (example ii) or questions (example iii) to alter the original answer, preventing the model from relying on memorized visual or linguistic patterns.
• Figure 3: Qualitative results. Between Image 1 and Image 2, both the camera position and the kettle position change, and the question asks how the kettle moves. With the blueprint serving as a well-structured representation, our model identifies the socket as a fixed reference and correctly infers the kettle’s movement. Other methods either miss the movement entirely or fail to account for the camera motion.
• Figure 4: Visualization of attention maps follows prior work Chen2025WhySpatialHard. In our method, high-relevance image patches cluster tightly around the true region of interest, whereas in other methods they tend to scatter.
• Figure 5: Example of extending our method to video-based spatial reasoning using a frame-selection module.