When Seeing Is not Enough: Revealing the Limits of Active Reasoning in MLLMs

TL;DR

GuessBench formalizes active multimodal reasoning as an interactive target-guessing task where MLLMs must actively acquire missing evidence under incomplete information. The framework combines a diverse dataset of real and synthetic images with a binary-question protocol and a composite metric that jointly weighs accuracy, reasoning efficiency, and planning. Across 20 mainstream MLLMs, results reveal substantial gaps between active and passive reasoning, with perception and timely decision-making as key bottlenecks; explicit thinking strategies consistently improve performance, especially for smaller models. The work identifies scalable, directionally promising paths such as perceptual enhancements for small models and thinking-oriented training across models, offering a clear blueprint for advancing multimodal active reasoning in real-world settings.

Abstract

Multimodal large language models (MLLMs) have shown strong capabilities across a broad range of benchmarks. However, most existing evaluations focus on passive inference, where models perform step-by-step reasoning under complete information. This setup is misaligned with real-world use, where seeing is not enough. This raises a fundamental question: Can MLLMs actively acquire missing evidence under incomplete information? To bridge this gap, we require the MLLMs to actively acquire missing evidence and iteratively refine decisions under incomplete information, by selecting a target image from a candidate pool without task-specific priors. To support systematic study, we propose GuessBench, a benchmark with both perception-oriented and knowledge-oriented images for evaluating active reasoning in MLLMs. We evaluate 20 superior MLLMs and find that performance on active reasoning lags far behind it on passive settings, indicating substantial room for improvement. Further analysis identifies fine-grained perception and timely decision-making as key challenges. Ablation studies show that perceptual enhancements benefit smaller models, whereas thinking-oriented methods provide consistent gains across model sizes. These results suggest promising directions for future research on multimodal active reasoning.

Figures (14)

• Figure 1: Passive vs. active reasoning. The left shows the difference between these two, and the right shows a pronounced gap between the two capabilities.
• Figure 2: Overview of data distribution, interaction pipeline, and metric design.
• Figure 3: The accuracy compared with the passive reasoning. The no-information baseline yields near-chance accuracy, confirming the reliability of our tasks.
• Figure 4: The difference between the original images (coarse-grained) and edited images (fine-grained).
• Figure 5: Accuracy and interaction length with different pool sizes.