Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

V-FAT: Benchmarking Visual Fidelity Against Text-bias

Ziteng Wang, Yujie He, Guanliang Li, Siqi Yang, Jiaqi Xiong, Songxiang Liu

TL;DR

V-FAT introduces a diagnostic framework to quantify visual fidelity under text bias in Multimodal LLMs by decoupling Internal Corpus Bias and External Instruction Bias. The Three-Level Evaluation Framework and Visual Robustness Score provide granular diagnostics that reveal substantial visual collapse under high linguistic dominance, with scaling alone not eliminating bias. The study finds notable differences between proprietary and open-source models, highlights a robustness gradient across bias levels, and identifies inference-time reasoning as a potential amplifier of textual bias. The results motivate direction shifts from mere scaling to training strategies that reinforce true visual grounding and resist linguistic priors in vision-language models.

Abstract

Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated impressive performance on standard visual reasoning benchmarks. However, there is growing concern that these models rely excessively on linguistic shortcuts rather than genuine visual grounding, a phenomenon we term Text Bias. In this paper, we investigate the fundamental tension between visual perception and linguistic priors. We decouple the sources of this bias into two dimensions: Internal Corpus Bias, stemming from statistical correlations in pretraining, and External Instruction Bias, arising from the alignment-induced tendency toward sycophancy. To quantify this effect, we introduce V-FAT (Visual Fidelity Against Text-bias), a diagnostic benchmark comprising 4,026 VQA instances across six semantic domains. V-FAT employs a Three-Level Evaluation Framework that systematically increases the conflict between visual evidence and textual information: (L1) internal bias from atypical images, (L2) external bias from misleading instructions, and (L3) synergistic bias where both coincide. We introduce the Visual Robustness Score (VRS), a metric designed to penalize "lucky" linguistic guesses and reward true visual fidelity. Our evaluation of 12 frontier MLLMs reveals that while models excel in existing benchmarks, they experience significant visual collapse under high linguistic dominance.

V-FAT: Benchmarking Visual Fidelity Against Text-bias

TL;DR

V-FAT introduces a diagnostic framework to quantify visual fidelity under text bias in Multimodal LLMs by decoupling Internal Corpus Bias and External Instruction Bias. The Three-Level Evaluation Framework and Visual Robustness Score provide granular diagnostics that reveal substantial visual collapse under high linguistic dominance, with scaling alone not eliminating bias. The study finds notable differences between proprietary and open-source models, highlights a robustness gradient across bias levels, and identifies inference-time reasoning as a potential amplifier of textual bias. The results motivate direction shifts from mere scaling to training strategies that reinforce true visual grounding and resist linguistic priors in vision-language models.

Abstract

Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated impressive performance on standard visual reasoning benchmarks. However, there is growing concern that these models rely excessively on linguistic shortcuts rather than genuine visual grounding, a phenomenon we term Text Bias. In this paper, we investigate the fundamental tension between visual perception and linguistic priors. We decouple the sources of this bias into two dimensions: Internal Corpus Bias, stemming from statistical correlations in pretraining, and External Instruction Bias, arising from the alignment-induced tendency toward sycophancy. To quantify this effect, we introduce V-FAT (Visual Fidelity Against Text-bias), a diagnostic benchmark comprising 4,026 VQA instances across six semantic domains. V-FAT employs a Three-Level Evaluation Framework that systematically increases the conflict between visual evidence and textual information: (L1) internal bias from atypical images, (L2) external bias from misleading instructions, and (L3) synergistic bias where both coincide. We introduce the Visual Robustness Score (VRS), a metric designed to penalize "lucky" linguistic guesses and reward true visual fidelity. Our evaluation of 12 frontier MLLMs reveals that while models excel in existing benchmarks, they experience significant visual collapse under high linguistic dominance.
Paper Structure (17 sections, 2 equations, 6 figures, 3 tables)

This paper contains 17 sections, 2 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Visual Hallucination Evaluation
  4. Text Bias and Language Priors
  5. V-FAT
  6. Benchmark Categories and Curation
  7. The Three-Level Evaluation Framework
  8. Visual Robustness Score
  9. Experiment
  10. Experimental Setup
  11. Experiment Analysis
  12. VRS by Levels
  13. Error Analysis
  14. Ablation
  15. Conclusion
  16. ...and 2 more sections

Figures (6)

  • Figure 1: Textual bias sources in MLLMs: (1) Internal Corpus Bias via pretraining correlations, and (2) External Instruction Bias via sycophancy to misleading prompts despite visual evidence.
  • Figure 2: Holistic Performance Evaluation. Radar charts illustrating the (a) Accuracy and (b) Visual Robustness Score (VRS) of 8 representative MLLMs across six metrics under three distinct bias levels (Level 1 - Level 3) for both Multiple-Choice (MCQ) and Open-Ended (OE) formats.
  • Figure 3: Hierarchical Diagnostic Protocol for Measuring Text Bias: This framework illustrates how MLLMs respond to escalating levels of linguistic interference. Level 1 identifies cases where internal pre-training associations override atypical visual facts; Level 2 isolates alignment-induced sycophancy when facing false premises; and Level 3 examines the compounding effect of dual-source textual conflict against objective visual reality.
  • Figure 4: Comprehensive VRS performance of proprietary and open-source MLLMs across three levels of textual bias. The "Robustness Gap" is clearly visible as textual pressure intensifies from Level 1 to Level 3.
  • Figure 5: MLLMs performances on different question types
  • ...and 1 more figures