Investigating Spatial Attention Bias in Vision-Language Models

TL;DR

Vision-Language Models exhibit a robust left-first spatial attention bias when describing content in horizontally concatenated images. The authors conduct controlled experiments across seven architectures using object-centric Caltech-101 pairs and dense Desktop UI contexts, with neutral and directional prompts, including an Arabic RTL model to test language priors. They find a near-universal left-to-right preference under neutral prompting, and show that language direction and explicit dataset annotations are unlikely causes, pointing toward architectural factors such as positional embeddings and vision encoder design. Structured prompting can mitigate but not fully eliminate the bias, underscoring the need for mechanistic interpretability and architecture-aware interventions to improve reliable, fair spatial processing in VLMs.

Abstract

Vision-Language Models have demonstrated remarkable capabilities in understanding visual content, yet systematic biases in their spatial processing remain largely unexplored. This work identifies and characterizes a systematic spatial attention bias where VLMs consistently prioritize describing left-positioned content before right-positioned content in horizontally concatenated images. Through controlled experiments on image pairs using both open-source and closed-source models, we demonstrate that this bias persists across different architectures, with models describing left-positioned content first in approximately 97% of cases under neutral prompting conditions. Testing on an Arabic-finetuned model reveals that the bias persists despite right-to-left language training, ruling out language reading direction as the primary cause. Investigation of training dataset annotation guidelines from PixMo and Visual Genome reveals no explicit left-first ordering instructions, suggesting the bias is consistent with architectural factors rather than explicit training data instructions. These findings reveal fundamental limitations in how current VLMs process spatial information.

Figures (1)

• Figure 1: Overview of the experimental setup for detecting spatial attention bias.