Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomalies

TL;DR

VIA-Bench is proposed as a diagnostic benchmark to stress-test multimodal large language models on visual illusions and anomalies across six categories, using 1,004 QA pairs formed through careful human-in-the-loop curation. The task formalization includes an illusory image, a question, candidate options, and a potentially CoT-enabled prompt, with outputs evaluated via $a = \frac{n}{N}$ under two protocols and ground-truth grounded in visual evidence. Across 20+ models, humans substantially outperform current MLLMs (93.30% vs 69.23%), and CoT reasoning often reduces robustness, revealing brittle reliance on internal priors rather than grounded perception. The work highlights persistent perceptual bottlenecks in state-of-the-art systems, offers qualitative analyses of failure modes, and commits to public release of data and code to spur future advances toward human-like multimodal perception.

Abstract

Multimodal Large Language Models (MLLMs) have shown remarkable proficiency on general-purpose vision-language benchmarks, reaching or even exceeding human-level performance. However, these evaluations typically rely on standard in-distribution data, leaving the robustness of MLLMs largely unexamined when faced with scenarios that defy common-sense priors. To address this gap, we introduce VIA-Bench, a challenging benchmark designed to probe model performance on visual illusions and anomalies. It includes six core categories: color illusions, motion illusions, gestalt illusions, geometric and spatial illusions, general visual illusions, and visual anomalies. Through careful human-in-the-loop review, we construct over 1K high-quality question-answer pairs that require nuanced visual reasoning. Extensive evaluation of over 20 state-of-the-art MLLMs, including proprietary, open-source, and reasoning-enhanced models, uncovers significant vulnerabilities. Notably, we find that Chain-of-Thought (CoT) reasoning offers negligible robustness, often yielding ``brittle mirages'' where the model's logic collapses under illusory stimuli. Our findings reveal a fundamental divergence between machine and human perception, suggesting that resolving such perceptual bottlenecks is critical for the advancement of artificial general intelligence. The benchmark data and code will be released.

Figures (12)

• Figure 1: Overview of VIA-Bench. The benchmark includes six categories: color illusions, motion illusions, gestalt illusions, geometric and spatial illusions, general visual illusions, and visual anomalies. These scenarios require MLLMs to develop human-like perception and deliberate reasoning. On VIA-Bench, humans achieve 93.30% average accuracy, whereas the best MLLM reaches 69.23%.
• Figure 2: Benchmark construction pipeline. The workflow progresses from data collection to unification, annotation, and debiasing, ultimately forming VIA-Bench. To ensure high quality, we apply human-in-the-loop assessment at all key stages.
• Figure 3: Statistical characterization of VIA-Bench. (a) Distribution across the six primary categories of illusions and anomalies. (b) Mapping of the dataset to specific perceptual and cognitive capabilities. (c) Empirical distribution of question lengths.
• Figure 4: Visualization of model responses on VIA-Bench. These examples demonstrate that even leading models struggle to handle relatively simple tasks such as counting, color recognition, and perceiving fine-grained detail on visual illusions and anomalies. $\text{-o3}$: OpenAI o3; : Gemini-2.5-pro; $\text{-o4}$: OpenAI o4; : Qwen3-VL-30B-A3B-Thinking. More cases can be found in the Appendix H.
• Figure 5: Relative gains of CoT over baseline prompting. Negative values indicate performance attenuation after applying CoT prompts. These results underscore that VIA-Bench presents a fundamental perceptual bottleneck that is not easily bypassed through surface-level textual prompting.