TopoPerception: A Shortcut-Free Evaluation of Global Visual Perception in Large Vision-Language Models

TL;DR

This work targets the global visual perception bottleneck in LVLMs and argues that existing benchmarks embed local shortcuts that inflate performance. It introduces TopoPerception, a topology-based, shortcut-free benchmark that uses fixed prompts and synthetic topological images across multiple granularity levels to probe global perception. Empirical results show state-of-the-art LVLMs perform near random at the coarsest level, with larger models sometimes doing worse, highlighting that scaling up reasoning does not fix perceptual fidelity. The authors discuss shortcuts, advocate synthetic data to isolate global structure, and call for new training paradigms or architectures to preserve global visual information; code and data are publicly available to facilitate further research.

Abstract

Large Vision-Language Models (LVLMs) typically align visual features from an encoder with a pre-trained Large Language Model (LLM). However, this makes the visual perception module a bottleneck, which constrains the overall capabilities of LVLMs. Conventional evaluation benchmarks, while rich in visual semantics, often contain unavoidable local shortcuts that can lead to an overestimation of models' perceptual abilities. Here, we introduce TopoPerception, a benchmark that leverages topological properties to rigorously evaluate the global visual perception capabilities of LVLMs across various granularities. Since topology depends on the global structure of an image and is invariant to local features, TopoPerception enables a shortcut-free assessment of global perception, fundamentally distinguishing it from semantically rich tasks. We evaluate state-of-the-art models on TopoPerception and find that even at the coarsest perceptual granularity, all models perform no better than random chance, indicating a profound inability to perceive global visual features. Notably, a consistent trend emerge within model families: more powerful models with stronger reasoning capabilities exhibit lower accuracy. This suggests that merely scaling up models is insufficient to address this deficit and may even exacerbate it. Progress may require new training paradigms or architectures. TopoPerception not only exposes a critical bottleneck in current LVLMs but also offers a lens and direction for improving their global visual perception. The data and code are publicly available at: https://github.com/Wenhao-Zhou/TopoPerception.

Figures (7)

• Figure 1: Accuracy of various LVLMs on the easiest level of TopoPerception. Models from the same family are represented by the same color scheme.
• Figure 2: An illustration of the TopoPerception benchmark. The text question and options are fixed (left). The input image belongs to one of three categories, corresponding to options B, C, and D, respectively (right). Options A and E serve as distractors. The input images can be extended to arbitrary difficulty levels.
• Figure 3: The visual topological images are generated by constructing a uniform spanning tree on a connected graph. Each node in the graph corresponds to a $3\times 3$ pixel block in the image. A connected graph with $n\times n$ nodes generates an image with a resolution of $(4n+1)\times (4n+1)$.
• Figure 4: Illustration of the granularity of image partitions at different levels in TopoPerception. The difficulty level for an image with a partitioning granularity of $(4n+1)\times (4n+1)$ is defined as $(n-7)/2$.
• Figure 5: An example of how the topological properties of an image in TopoPerception change across adjacent perceptual granularities.