How Much Information Can a Vision Token Hold? A Scaling Law for Recognition Limits in VLMs

TL;DR

This work investigates the fundamental information capacity of vision tokens in vision-language models by conducting controlled experiments with dense text rendered as images under fixed token budgets. It reveals a scale-invariant phase transition with three regimes—Stable, Instability, and Collapse—and distinguishes reversible spatial-alignment failures from irreversible capacity exhaustion. A probabilistic scaling law is introduced, combining visual density $V$ and average token load $G$ through $Z = w_0 + a \log V + \alpha \log G$ and a mixture model, which is shown to generalize across multiple VLM architectures. The findings offer practical guidance for compression-aware VLM design and adaptive inference to optimize efficiency-accuracy trade-offs in long-context visual contexts.

Abstract

Recent vision-centric approaches have made significant strides in long-context modeling. Represented by DeepSeek-OCR, these models encode rendered text into continuous vision tokens, achieving high compression rates without sacrificing recognition precision. However, viewing the vision encoder as a lossy channel with finite representational capacity raises a fundamental question: what is the information upper bound of visual tokens? To investigate this limit, we conduct controlled stress tests by progressively increasing the information quantity (character count) within an image. We observe a distinct phase-transition phenomenon characterized by three regimes: a near-perfect Stable Phase, an Instability Phase marked by increased error variance, and a total Collapse Phase. We analyze the mechanical origins of these transitions and identify key factors. Furthermore, we formulate a probabilistic scaling law that unifies average vision token load and visual density into a latent difficulty metric. Extensive experiments across various Vision-Language Models demonstrate the universality of this scaling law, providing critical empirical guidance for optimizing the efficiency-accuracy trade-off in visual context compression.

Figures (15)

• Figure 1: Illustration of the Block-wise Shuffling strategy. Text blocks are segmented, randomly sampled, and concatenated to construct randomized semantic text.
• Figure 2: Scatter plots of ED versus Text Length (TL) across four resolutions ($R$). Three distinct regimes (shaded) emerge across semantic domains: Stable Phase (green), Instability Phase (yellow), and Collapse Phase (gray). Higher resolutions exhibit a wider Instability Phase and shift the hard wall to the right.
• Figure 3: Results of the Pixel-Shift Perturbation experiment. We plot ED against Text Length (TL) for resolutions $R=640$ and $R=1024$. The black vertical dashed line marks the Hard Wall separating Zone I from Zone II. The yellow vertical lines connect the original performance (green squares) to the minimum ED (red triangles) achieved after perturbation for the same sample. The visualization demonstrates that errors in Zone I are reversible through spatial alignment, whereas errors in Zone II persist regardless of pixel shifting.
• Figure 4: Visual density alignment on Novels: 1024 Stable denotes Stable-group samples (TL < 5k) evaluated at input resolution $R=1024$ (analogously for other labels). Despite matched character scale, Collapse-group samples (TL > 15k) retain high ED, supporting an information-capacity limit.
• Figure 5: Alignment of performance curves in the Latent Difficulty space. The alignment of data points from different resolutions confirms the scaling law. The vertical boundaries show that the transition thresholds into Zone I and Zone II are identical across all configurations when plotted against $Z$.