I Spy With My Model's Eye: Visual Search as a Behavioural Test for MLLMs

TL;DR

This work investigates how multimodal large language models perceive visual scenes by treating vision as a cognitive-like process. It adapts classic visual-search paradigms to probe pop-out effects and feature binding in MLLMs, comparing them to human baselines. The authors report that top-tier models exhibit human-like disjunctive search with color/size salience and show capacity limits under conjunctive search, and that lighting priors are incorporated into object representations. They further demonstrate that fine-tuning can improve conjunctive-search performance and that mechanistic interpretability reveals layerwise distinctions between disjunctive and conjunctive processing. The findings suggest visual-search-based diagnostics can reveal perceptual capabilities and biases in MLLMs, with implications for design, evaluation, and safe deployment.

Abstract

Multimodal large language models (MLLMs) achieve strong performance on vision-language tasks, yet their visual processing is opaque. Most black-box evaluations measure task accuracy, but reveal little about underlying mechanisms. Drawing on cognitive psychology, we adapt classic visual search paradigms -- originally developed to study human perception -- to test whether MLLMs exhibit the ``pop-out'' effect, where salient visual features are detected independently of distractor set size. Using controlled experiments targeting colour, size and lighting features, we find that advanced MLLMs exhibit human-like pop-out effects in colour or size-based disjunctive (single feature) search, as well as capacity limits for conjunctive (multiple feature) search. We also find evidence to suggest that MLLMs, like humans, incorporate natural scene priors such as lighting direction into object representations. We reinforce our findings using targeted fine-tuning and mechanistic interpretability analyses. Our work shows how visual search can serve as a cognitively grounded diagnostic tool for evaluating perceptual capabilities in MLLMs.

Figures (20)

• Figure 1: Circle Sizes task. Examples from the three experimental conditions. The target is always the single circle that is larger than the rest. The colour of all circles is always the same and is sampled from red, blue and green.
• Figure 2: Results for Circle Sizes on Cells.
• Figure 3: Example stimuli from the 2 Among 5 task, illustrating the three experimental conditions. pronounced In all examples, the target "2" is coloured red. (a) Disjunctive: The target differs from distractors by colour. (b) Shape Conjunctive: All digits share the same colour, requiring shape discrimination. (c) Shape-Colour Conjunctive: The target is uniquely defined by both shape and colour, with distractors sharing at most one feature. Target and distractor colours are randomized across trials.
• Figure 4: Results for the 2Among5 task — Cells mode. The shaded region denotes the 95% confidence interval.
• Figure 5: Light Priors Task: Circles are shaded with directional gradients, mimicking the shading on a sphere if lit from different directions. The subject must identify the target lit from a specific direction.