VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs

TL;DR

VisRes tackles the question of true visual reasoning in vision–language systems by introducing a real-image, four-choice benchmark with three progressively demanding levels that separate perceptual grounding, single-attribute rule inference, and multi-attribute compositional reasoning. The benchmark, evaluated with guided prompting and thinking-mode across multiple SOTA systems, reveals systematic deficits: perceptual degradation drastically reduces performance, single-attribute tasks are more tractable (especially color), while multi-attribute composition remains challenging, and a clear modality gap exists between vision-based and text-based reasoning. Supervised fine-tuning partially closes the gap on Level-1 but human performance remains far superior; higher image resolution helps but does not fix core perceptual-grounding and reasoning bottlenecks. VisRes offers a principled framework to diagnose and guide future architectural improvements that integrate robust perception with abstract reasoning in multimodal settings.

Abstract

Vision-Language Models (VLMs) have achieved remarkable progress across tasks such as visual question answering and image captioning. Yet, the extent to which these models perform visual reasoning as opposed to relying on linguistic priors remains unclear. To address this, we introduce VisRes Bench, a benchmark designed to study visual reasoning in naturalistic settings without contextual language supervision. Analyzing model behavior across three levels of complexity, we uncover clear limitations in perceptual and relational visual reasoning capacities. VisRes isolates distinct reasoning abilities across its levels. Level 1 probes perceptual completion and global image matching under perturbations such as blur, texture changes, occlusion, and rotation; Level 2 tests rule-based inference over a single attribute (e.g., color, count, orientation); and Level 3 targets compositional reasoning that requires integrating multiple visual attributes. Across more than 19,000 controlled task images, we find that state-of-the-art VLMs perform near random under subtle perceptual perturbations, revealing limited abstraction beyond pattern recognition. We conclude by discussing how VisRes provides a unified framework for advancing abstract visual reasoning in multimodal research.

Figures (15)

• Figure 1: Real samples from each level. Level 1 (top) involves direct visual completion and matching without explicit rule inference (e.g., patch-C correctly continues the ceiling texture compared to patch-D), while Levels 2 and 3 (bottom) require increasingly complex rule-based reasoning over perceptual attributes. Accurate perception of individual attributes is necessary but not sufficient for solving compositional tasks. Current VLMs show poor performance on these compositional tasks. See Section \ref{['sec:performance']}.
• Figure 2: Illustrative pattern rules used in Levels 2 and 3 tasks. Top: Level-2 tasks where one attribute varies across the row. Bottom: Level-3 tasks where multiple attributes vary. See Section \ref{['sec:task_gen']}.
• Figure A.1: Examples of Level-1 perceptual completion tasks. Each task masks a local region of the main image, and the model must infer the missing content by comparing visual evidence around the blanked area with four candidate patches. The eight task variants isolate different perceptual cues: Location (RS), Location (DS), Edges (RS), Edges (DS), Blur (DS), Brightness (DS), Rotation (DS), Rotation (SP).
• Figure A.2: Global occlusion tasks in Level-1, where 50–80% of the scene is masked. Unlike local completion, the model must reason about the broader structure and context of the scene to recover the missing content. Only one candidate image (A–D) corresponds to the original, unoccluded scene.
• Figure A.3: Level-1 Location (DS): Prompt and Model Response Example