Do VLMs Perceive or Recall? Probing Visual Perception vs. Memory with Classic Visual Illusions

TL;DR

Do large vision–language systems truly ground perception or rely on memorized priors when facing classic visual illusions? VI-Probe provides a controllable, perturbation-based framework with graded visual changes, matched controls, and prompting variants, paired with metrics $PFC$, $PFA$, $TFI$, and $R$ to separate perception from memory. Across 15 families, the study reveals heterogeneous mechanisms—memory override, perception–memory competition, and visual-processing limits—challenging single-cause explanations and highlighting the need for counterfactual consistency and perception-first designs. The work offers practical evaluation guidance and will drive future counterfactual-aware training, with data and code released for reproducibility and extension to real-world visual judgments.

Abstract

Large Vision-Language Models (VLMs) often answer classic visual illusions "correctly" on original images, yet persist with the same responses when illusion factors are inverted, even though the visual change is obvious to humans. This raises a fundamental question: do VLMs perceive visual changes or merely recall memorized patterns? While several studies have noted this phenomenon, the underlying causes remain unclear. To move from observations to systematic understanding, this paper introduces VI-Probe, a controllable visual-illusion framework with graded perturbations and matched visual controls (without illusion inducer) that disentangles visually grounded perception from language-driven recall. Unlike prior work that focuses on averaged accuracy, we measure stability and sensitivity using Polarity-Flip Consistency, Template Fixation Index, and an illusion multiplier normalized against matched controls. Experiments across different families reveal that response persistence arises from heterogeneous causes rather than a single mechanism. For instance, GPT-5 exhibits memory override, Claude-Opus-4.1 shows perception-memory competition, while Qwen variants suggest visual-processing limits. Our findings challenge single-cause views and motivate probing-based evaluation that measures both knowledge and sensitivity to controlled visual change. Data and code are available at https://sites.google.com/view/vi-probe/.

Figures (33)

• Figure 1: Motivation for developing VI-Probe. On classic visual illusions (e.g., Ebbinghaus weintraub1979ebbinghaus), large VLMs often score well on Original images yet fail to flip along with GT on Perturbed images (factor inverted), even when the change is perceptually obvious to humans. Prior evaluations typically report average accuracy and often do not expose the Original$\rightarrow$Perturbed gap or explain its cause. A few studies have noted this discrepancy, but without deeper analysis. We propose VI-Probe to decompose model behavior and attribute the gap to potential contributing factors.
• Figure 2: Examples, vision and language variations in VI-Probe. For each original illusion case, we have six versions of images (perturbed-control is not shown in the figure), in which perturbed, perturbed-control and perturbed-hints have a series of images of different perturbation strength. For language, we have three versions of prompting questions.
• Figure 3: Distribution of perturbation images at different strengths. 2D PCA Visualizing embeddings from (A) Qwen2.5-VL-72B and (B) -3B, with color coding for perturbation strength. Left shows clear separation between different perturbation levels, indicating the validity of the data generation pipeline.
• Figure 4: Polarity–Flip Consistency (PFC) decomposition across VLMs. Each stacked bar partitions paired responses into: PFA = correct and complementary (both answers correct), CbW = complementary but not fully correct, and TFI = non–complementary. Models are ordered by overall consistency, $\mathrm{PFC}=\mathrm{PFA}+\mathrm{CbW}$; the dashed line at $50\%$ marks random.
• Figure 5: Model performance under controlled perturbations. (A) Accuracy vs. perturbation strength for two representative models (GPT-5 and Claude-Opus-4.1) under perturbed–control (vision-only) and perturbed–illusion (factor-inverted) settings; the gap between the two curves reflects memory vs. vision. (B) Heatmap for perturbed–control; (C) heatmap for perturbed–illusion. Columns denote perturbation strength (larger value is stronger). Accuracy typically decreases with strength, yet rankings diverge across (B) and (C). Red vertical lines indicate the threshold at which human observers reliably detect visual changes.