Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?

Qing'an Liu, Juntong Feng, Yuhao Wang, Xinzhe Han, Yujie Cheng, Yue Zhu, Haiwen Diao, Yunzhi Zhuge, Huchuan Lu

TL;DR

VISTA-Bench addresses whether vision-language models can match pure-text understanding when language is rendered as visualized text. It presents a rigorously controlled, 1,500-sample benchmark with a LaTeX-based rendering pipeline and a VLM-based fidelity filter to compare VT and text inputs across perception, reasoning, and knowledge tasks. Across more than 20 VLMs, the study uncovers a pervasive modality gap that worsens with perceptual difficulty, suggesting that strong pure-text performance does not guarantee VT robustness. The results highlight perceptual robustness as the core bottleneck and advocate for unified, robust representations that bridge tokenized text and pixel-based language.

Abstract

Vision-Language Models (VLMs) have achieved impressive performance in cross-modal understanding across textual and visual inputs, yet existing benchmarks predominantly focus on pure-text queries. In real-world scenarios, language also frequently appears as visualized text embedded in images, raising the question of whether current VLMs handle such input requests comparably. We introduce VISTA-Bench, a systematic benchmark from multimodal perception, reasoning, to unimodal understanding domains. It evaluates visualized text understanding by contrasting pure-text and visualized-text questions under controlled rendering conditions. Extensive evaluation of over 20 representative VLMs reveals a pronounced modality gap: models that perform well on pure-text queries often degrade substantially when equivalent semantic content is presented as visualized text. This gap is further amplified by increased perceptual difficulty, highlighting sensitivity to rendering variations despite unchanged semantics. Overall, VISTA-Bench provides a principled evaluation framework to diagnose this limitation and to guide progress toward more unified language representations across tokenized text and pixels. The source dataset is available at https://github.com/QingAnLiu/VISTA-Bench.

VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?

TL;DR

VISTA-Bench addresses whether vision-language models can match pure-text understanding when language is rendered as visualized text. It presents a rigorously controlled, 1,500-sample benchmark with a LaTeX-based rendering pipeline and a VLM-based fidelity filter to compare VT and text inputs across perception, reasoning, and knowledge tasks. Across more than 20 VLMs, the study uncovers a pervasive modality gap that worsens with perceptual difficulty, suggesting that strong pure-text performance does not guarantee VT robustness. The results highlight perceptual robustness as the core bottleneck and advocate for unified, robust representations that bridge tokenized text and pixel-based language.

Abstract

Vision-Language Models (VLMs) have achieved impressive performance in cross-modal understanding across textual and visual inputs, yet existing benchmarks predominantly focus on pure-text queries. In real-world scenarios, language also frequently appears as visualized text embedded in images, raising the question of whether current VLMs handle such input requests comparably. We introduce VISTA-Bench, a systematic benchmark from multimodal perception, reasoning, to unimodal understanding domains. It evaluates visualized text understanding by contrasting pure-text and visualized-text questions under controlled rendering conditions. Extensive evaluation of over 20 representative VLMs reveals a pronounced modality gap: models that perform well on pure-text queries often degrade substantially when equivalent semantic content is presented as visualized text. This gap is further amplified by increased perceptual difficulty, highlighting sensitivity to rendering variations despite unchanged semantics. Overall, VISTA-Bench provides a principled evaluation framework to diagnose this limitation and to guide progress toward more unified language representations across tokenized text and pixels. The source dataset is available at https://github.com/QingAnLiu/VISTA-Bench.
Paper Structure (40 sections, 19 figures, 6 tables)

This paper contains 40 sections, 19 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Understanding Benchmarks
  4. Emerging Paradigms in Visualized Text
  5. VISTA-Bench
  6. Task Settings
  7. Preliminary experiments
  8. VISTA-Bench Construction
  9. Experiments
  10. Setup
  11. Main Results
  12. Fine-grained Analysis
  13. Limitations and Disscusion
  14. Conclusion
  15. Benchmark Details
  16. ...and 25 more sections

Figures (19)

  • Figure 1: (a) Humans integrate visual context with embedded text, whereas standard VLM evaluation provides language as discrete tokens. (b) Presenting language as visualized text can induce behavioral deviations from pure-text inputs, revealing a modality gap.
  • Figure 2: Comparison between Text and Visualized Text Inputs.
  • Figure 3: Perceptual factor impact.Top: Font Size (9, 16, 32, 48, 64). Bottom: Font Style (Arial, Cambria, Roman, Brush).
  • Figure 4: Overview of the construction. First, we extract filtered dataset from existing data rely on diversity and accuracy. Second, we transform text into visualized text through the rendering pipeline. We then validate the precision of visualized text depends on VLM and continuously refine the pipeline. Through this process, we finally establish VISTA-Bench, supported by a sophisticated rendering pipeline.
  • Figure 5: Ability dimensions in VISTA-Bench. VISTA-Bench includes two main levels of dimensions based on Inherent Modality Dependence and Cognitive Dimension, with 10 distinct abilities.
  • ...and 14 more figures