Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs

TL;DR

The paper addresses the binding problem in LVLMs, where parallel visual processing leads to misbinding of features to objects, degrading counting, visual search, scene description, and spatial reasoning. It introduces Visual Input Structure for Enhanced Reasoning (VISER), which augments images with simple horizontal lines and pairs them with a sequential scanning prompt to promote region-wise, serial parsing. Across synthetic and real-world benchmarks and a mix of LVLMs, VISER yields substantial gains on core visual reasoning tasks (e.g., approximately 25–27% improvements in visual search and counting, about a 9.5% gain in spatial reasoning, and a 0.32 reduction in edit distance for scene descriptions), while purely textual strategies like CoT offer limited or negative gains. The results underscore the importance of visual input design for binding and reasoning, suggesting avenues for adaptive scaffolds and integrated spatial attention in future work to further mitigate binding errors.

Abstract

Despite progress in Large Vision-Language Models (LVLMs), their capacity for visual reasoning is often limited by the binding problem: the failure to reliably associate perceptual features with their correct visual referents. This limitation underlies persistent errors in tasks such as counting, visual search, scene description, and spatial relationship understanding. A key factor is that current LVLMs process visual features largely in parallel, lacking mechanisms for spatially grounded, serial attention. This paper introduces Visual Input Structure for Enhanced Reasoning (VISER), a simple, effective method that augments visual inputs with low-level spatial structures and pairs them with a textual prompt that encourages sequential, spatially-aware parsing. We empirically demonstrate substantial performance improvements across core visual reasoning tasks, using only a single-query inference. Specifically, VISER improves GPT-4o performance on visual search, counting, and spatial relationship tasks by 25.0%, 26.8%, and 9.5%, respectively, and reduces edit distance error in scene description by 0.32 on 2D datasets. Furthermore, we find that the visual modification is essential for these gains; purely textual strategies, including Chain-of-Thought prompting, are insufficient and can even degrade performance. VISER underscores the importance of visual input design over purely linguistically based reasoning strategies and suggests that visual structuring is a powerful and general approach for enhancing compositional and spatial reasoning in LVLMs.

Figures (29)

• Figure 1: The input image is augmented with low-level visual structure using three horizontal lines, optionally accompanied by row annotations on the left side of the image. A corresponding textual prompt (“Scan the image sequentially based on horizontal lines”) is appended to encourage the model to adopt a spatially guided, sequential parsing strategy.
• Figure 2: A brief summary of tasks with one example of synthetic data along the specific prompt for each task. (a) Visual Search, (b) Scene Description, (c) Counting, and (d) Spatial Relationship.
• Figure 3: Comparison of VISER with Chain-of-Thought (CoT) prompting on the GPT-4o model across four tasks. Each subplot shows the performance of three methods (Baseline, CoT, and VISER) evaluated using a task-specific metric indicated on the x-axis. Bars are grouped into 2D and 3D datasets.
• Figure 4: Comparison of VISER with visual reasoning–finetuned model on the Qwen2.5-VL base model across four tasks. Each subplot shows the performance of four methods: Qwen-Baseline (Qwen2.5-VL), Qwen-VISER (VISER applied to Qwen2.5-VL), Mulberry (Qwen2.5-VL finetuned for visual reasoning), and OpenVLThinker (RL-finetuned Qwen2.5-VL), evaluated using task-specific metrics, and results are grouped into 2D and 3D datasets.
• Figure 5: Performance of GPT-4o and Qwen2.5-VL across different tasks (Counting, Visual Search, and Scene Description) with varying numbers of horizontal lines in the input. Accuracy is reported for Counting and Visual Search, while edit distance is used for Scene Description. Baseline represents performance with no horizontal lines.