VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge

TL;DR

VisualPuzzles presents a knowledge-light, multimodal reasoning benchmark with 1,168 items across five reasoning categories to isolate general inference from domain-specific knowledge. The authors demonstrate that current multimodal LLMs underperform humans even with CoT prompting, and that larger models do not consistently improve reasoning on this task. They show VisualPuzzles has lower knowledge intensity yet higher reasoning complexity than knowledge-heavy benchmarks like MMMU, revealing a different bottleneck in multimodal reasoning. The work highlights the need for architectures and training strategies that emphasize structured reasoning over mere scale or memorized knowledge, and it opens avenues for broader evaluation formats and real-world applicability. Overall, VisualPuzzles provides a clearer lens to assess true multimodal reasoning capabilities beyond factual recall.

Abstract

Current multimodal benchmarks often conflate reasoning with domain-specific knowledge, making it difficult to isolate and evaluate general reasoning abilities in non-expert settings. To address this, we introduce VisualPuzzles, a benchmark that targets visual reasoning while deliberately minimizing reliance on specialized knowledge. VisualPuzzles consists of diverse questions spanning five categories: algorithmic, analogical, deductive, inductive, and spatial reasoning. One major source of our questions is manually translated logical reasoning questions from the Chinese Civil Service Examination. Experiments show that VisualPuzzles requires significantly less intensive domain-specific knowledge and more complex reasoning compared to benchmarks like MMMU, enabling us to better evaluate genuine multimodal reasoning. Evaluations show that state-of-the-art multimodal large language models consistently lag behind human performance on VisualPuzzles, and that strong performance on knowledge-intensive benchmarks does not necessarily translate to success on reasoning-focused, knowledge-light tasks. Additionally, reasoning enhancements such as scaling up inference compute (with "thinking" modes) yield inconsistent gains across models and task types, and we observe no clear correlation between model size and performance. We also found that models exhibit different reasoning and answering patterns on VisualPuzzles compared to benchmarks with heavier emphasis on knowledge. VisualPuzzles offers a clearer lens through which to evaluate reasoning capabilities beyond factual recall and domain knowledge.

Figures (43)

• Figure 1: Model accuracy on VisualPuzzles compared to human performance percentiles. All evaluated models fall below the human 5th percentile (57.5%), highlighting the difficulty of VisualPuzzles. Interestingly, models with explicit "thinking" modes do not consistently outperform their base versions, suggesting that current reasoning strategies do not yet generalize well to VisualPuzzles's scenarios, even though these strategies have proven effective in existing reasoning tasks that often rely heavily on domain-specific knowledge.
• Figure 2: Example VisualPuzzles instances within each reasoning category
• Figure 3: Scatter plots with trend lines of the relationship between accuracy and model size (top) and the relationship between reasoning and knowledge accuracy (bottom) on MMMU and VisualPuzzles. The dots' sizes represent relative model sizes. The correlation between reasoning accuracy and knowledge accuracy is higher on MMMU (0.8) than on VisualPuzzles (0.4).
• Figure 4: Comparison of accuracy and average number of total completion tokens of reasoning models and their general counterparts on VisualPuzzles. We didn't include Gemini-2.0-Flash models here because Gemini-2.0-Flash-Thinking does not reveal the number of reasoning tokens of responses. The accuracies of Gemini-2.0-Flash and Gemini-2.0-Flash-Thinking is 45.0% and 42.2% respectively. Despite much higher number of completion tokens, reasoning models do not often achieve better performance on VisualPuzzles.
• Figure 5: Comparison of Reasoning Pattern of Claude-3.7-Sonnet-Thinking on MMMU and VisualPuzzles. Left figure compares the accuracy of Claude-3.7-Sonnet and Claude-3.7-Sonnet-Thinking on MMMU and VisualPuzzles. Middle figure shows frequency of each pattern. Right figure shows correlation of the patterns with accuracy on the benchmarks.