Reasoning or Pattern Matching? Probing Large Vision-Language Models with Visual Puzzles

TL;DR

This work advocates visual puzzles as diagnostic instruments to evaluate large vision-language models' reasoning, separating abstract cognitive operations from perception and world knowledge. It introduces a unified abstraction $⟨I,R,S⟩$ and a five-type taxonomy (inductive, analogical, algorithmic, deductive, geometric) to organize benchmarks and compare model behavior across tasks. Across inductive, analogical, algorithmic, deductive, and geometric/d spatial domains, the survey uncovers consistent weaknesses—brittle generalization, perception–reasoning bottlenecks, reliance on superficial cues, and a disjunction between fluent explanations and faithful execution—arguing that current performance often reflects pattern matching rather than robust reasoning. The paper concludes with a roadmap for future benchmarks and reasoning-aware systems, emphasizing verification of intermediate states, compositional generalization, and collaborative multi-agent and abductive approaches to push LVLMs toward trustworthy, structured reasoning.

Abstract

Puzzles have long served as compact and revealing probes of human cognition, isolating abstraction, rule discovery, and systematic reasoning with minimal reliance on prior knowledge. Leveraging these properties, visual puzzles have recently emerged as a powerful diagnostic tool for evaluating the reasoning abilities of Large Vision-Language Models (LVLMs), offering controlled, verifiable alternatives to open-ended multimodal benchmarks. This survey provides a unified perspective of visual puzzle reasoning in LVLMs. We frame visual puzzles through a common abstraction and organize existing benchmarks by the reasoning mechanisms they target (inductive, analogical, algorithmic, deductive, and geometric/spatial), thereby linking puzzle design to the cognitive operations required for solving. Synthesizing empirical evidence across these categories, we identify consistent limitations in current models, including brittle generalization, tight entanglement between perception and reasoning, and a persistent gap between fluent explanations and faithful execution. By framing visual puzzles as diagnostic instruments rather than task formats, this survey elaborates on the state of LVLM reasoning and outlines key directions for future benchmarks and reasoning-aware multimodal systems.