VRIQ: Benchmarking and Analyzing Visual-Reasoning IQ of VLMs
Tina Khezresmaeilzadeh, Jike Zhong, Konstantinos Psounis
TL;DR
VRIQ introduces a diagnostic benchmark to evaluate visual reasoning IQ in vision-language models by pairing abstract and natural puzzles with hierarchical perceptual and reasoning probes. It enables end-to-end assessment and fine-grained error attribution, revealing that perception bottlenecks, not pure reasoning, largely constrain performance. Tool-augmented reasoning substantially improves results on several tasks, especially in natural imagery, though abstract puzzles remain difficult and human performance remains far higher. The benchmark provides actionable diagnostics and a framework to guide perceptual grounding and multimodal system design. Overall, VRIQ offers a principled way to quantify and diagnose visual reasoning in multimodal models and to steer future improvements.
Abstract
Recent progress in Vision Language Models (VLMs) has raised the question of whether they can reliably perform nonverbal reasoning. To this end, we introduce VRIQ (Visual Reasoning IQ), a novel benchmark designed to assess and analyze the visual reasoning ability of VLMs. We evaluate models on two sets of tasks: abstract puzzle-style and natural-image reasoning tasks. We find that on abstract puzzles, performance remains near random with an average accuracy of around 28%, while natural tasks yield better but still weak results with 45% accuracy. We also find that tool-augmented reasoning demonstrates only modest improvements. To uncover the source of this weakness, we introduce diagnostic probes targeting perception and reasoning. Our analysis demonstrates that around 56% of failures arise from perception alone, 43% from both perception and reasoning, and only a mere 1% from reasoning alone. This motivates us to design fine-grained diagnostic probe questions targeting specific perception categories (e.g., shape, count, position, 3D/depth), revealing that certain categories cause more failures than others. Our benchmark and analysis establish that current VLMs, even with visual reasoning tools, remain unreliable abstract reasoners, mostly due to perception limitations, and offer a principled basis for improving visual reasoning in multimodal systems.