Pursuing Minimal Sufficiency in Spatial Reasoning

TL;DR

Spatial reasoning in Vision-Language Models is hampered by limited 3D understanding and information redundancy. The paper introduces MSSR, a dual-agent framework that constructs a Minimal Sufficient Set ($S^ op$) before answering a query, using a Perception Agent with 3D reconstruction and a Reasoning Agent that prunes information and iterates through missing items, grounded by the SOG module. MSSR achieves state-of-the-art results on MMSI-Bench and ViewSpatial-Bench and yields interpretable reasoning traces. The MSS-based, zero-shot design preserves the model's general capabilities while providing high-quality data for future training.

Abstract

Spatial reasoning, the ability to ground language in 3D understanding, remains a persistent challenge for Vision-Language Models (VLMs). We identify two fundamental bottlenecks: inadequate 3D understanding capabilities stemming from 2D-centric pre-training, and reasoning failures induced by redundant 3D information. To address these, we first construct a Minimal Sufficient Set (MSS) of information before answering a given question: a compact selection of 3D perception results from \textit{expert models}. We introduce MSSR (Minimal Sufficient Spatial Reasoner), a dual-agent framework that implements this principle. A Perception Agent programmatically queries 3D scenes using a versatile perception toolbox to extract sufficient information, including a novel SOG (Situated Orientation Grounding) module that robustly extracts language-grounded directions. A Reasoning Agent then iteratively refines this information to pursue minimality, pruning redundant details and requesting missing ones in a closed loop until the MSS is curated. Extensive experiments demonstrate that our method, by explicitly pursuing both sufficiency and minimality, significantly improves accuracy and achieves state-of-the-art performance across two challenging benchmarks. Furthermore, our framework produces interpretable reasoning paths, offering a promising source of high-quality training data for future models. Source code is available at https://github.com/gyj155/mssr.

Figures (11)

• Figure 1: Irrelevant (highlighted in red) information can overwhelm the VLM and hurt its performance. Left: VLM ignores crucial information and hallucinates. Right: VLM cannot attend to the important part.
• Figure 2: The MSSR dual-agent framework. Top: To avoid overwhelming the VLM, MSSR extracts a Minimal Sufficient Set that only contains the necessary information. Bottom: Dual-agent loop: the Perception Agent provides a spatial information set ($S_n$), and the Reasoning Agent prunes irrelevant information, requests missing information, or makes a final decision based on the curated set.
• Figure 3: A detailed example. The Perception Agent provides an 18-item set $S_n$, and the Reasoning Agent prunes to 3 essential items in $S_{n+1}$. Deeming this insufficient, the RA issues a <Request> for missing calculations. With sufficient information, it makes a final <Decision>.
• Figure 4: Situated Orientation Grounding as a multi-choice question. We prompt a VLM to select a direction from multiple candidates using the original image and a rendered image, both overlayed with the candidate's 3D orientation vectors. Starting from coarse directions, it then refines this direction to a fine-grained result.
• Figure 5: Effects of conciseness on accuracy.