Reading Images Like Texts: Sequential Image Understanding in Vision-Language Models

TL;DR

The paper analyzes vision-language models by separating object perception (the what) from spatial reasoning (the where), revealing a two-stage object-recognition process in the visual encoder and a geometric interpretation of 2D RoPE for spatial relations. It introduces a logit-lens based method to map visual tokens to text, yielding token and segmentation maps that reveal how object labels emerge across ViT layers, and demonstrates that spatial relations can be captured through RoPE-derived geometry. To improve efficiency and reasoning, the authors propose an instruction-agnostic token compression using a visual decoder with Run-Length Encoding, and RoPE scaling to amplify low-frequency positional signals, validated on multiple benchmarks. Together, these insights offer principled guidance for building more capable, efficient, and interpretable VLM architectures. The work also provides reproducibility materials and empirical evidence linking layer-wise processing to Gestalt-inspired perceptual principles.” ,

Abstract

Vision-Language Models (VLMs) have demonstrated remarkable performance across a variety of real-world tasks. However, existing VLMs typically process visual information by serializing images, a method that diverges significantly from the parallel nature of human vision. Moreover, their opaque internal mechanisms hinder both deeper understanding and architectural innovation. Inspired by the dual-stream hypothesis of human vision, which distinguishes the "what" and "where" pathways, we deconstruct the visual processing in VLMs into object recognition and spatial perception for separate study. For object recognition, we convert images into text token maps and find that the model's perception of image content unfolds as a two-stage process from shallow to deep layers, beginning with attribute recognition and culminating in semantic disambiguation. For spatial perception, we theoretically derive and empirically verify the geometric structure underlying the positional representation in VLMs. Based on these findings, we introduce an instruction-agnostic token compression algorithm based on a plug-and-play visual decoder to improve decoding efficiency, and a RoPE scaling technique to enhance spatial reasoning. Through rigorous experiments, our work validates these analyses, offering a deeper understanding of VLM internals and providing clear principles for designing more capable future architectures.

Figures (24)

• Figure 1: The typical architecture of the vision-language model. It consists of an visual encoder, a modality connector and a decoder-only language model. Notations are detailed in Section \ref{['sec: vlm arch']}
• Figure 2: An illustration of token map.
• Figure 3: The segmentation maps of the image representations in ViT layers of LLaVA-1.5-7B. From shallow to deep layers, ViT gradually performs semantic disambiguation.
• Figure 4: The change in the ratio of attribute words and representative words with ViT layers in LLaVA.
• Figure 5: The geometry of the 1D absolute position embedding in LLaVA-1.5-7B and InternVL-2.5-8B (w/o [CLS] token). The visualization is performed via t-SNE for dimensionality reduction. The labels are set to row IDs (\ref{['fig: llava_pos_embed_row']}, \ref{['fig: intern_pos_embed_row']}) and column IDs (\ref{['fig: llava_pos_embed_col']}, \ref{['fig: intern_pos_embed_col']}), respectively.