Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LanP: Rethinking the Impact of Language Priors in Large Vision-Language Models

Zongyu Wu, Yuwei Niu, Hongcheng Gao, Minhua Lin, Zhiwei Zhang, Zhifang Zhang, Qi Shi, Yilong Wang, Sike Fu, Junjie Xu, Junjie Ao, Enyan Dai, Lei Feng, Xiang Zhang, Suhang Wang

TL;DR

LanP introduces a targeted benchmark to reassess the positive role of language priors in LVLMs, emphasizing how priors can aid or hinder answers under visual ambiguity. Through 170 images and 340 carefully crafted yes/no questions across four categories, the study evaluates 25 LVLMs and finds that many models still rely insufficiently on language priors to resolve partially hidden objects. The results show a clear size-and-architecture dependency: larger LLMs and certain configurations can leverage priors to improve answers, yet Partially Hidden cases remain challenging, suggesting a need for balanced priors. Overall, LanP provides a framework for diagnosing and guiding the development of LVLMs that effectively fuse world knowledge with visual signals, with implications for reducing hallucination while maintaining reasoning capabilities.

Abstract

Large Vision-Language Models (LVLMs) have shown impressive performance in various tasks. However, LVLMs suffer from hallucination, which hinders their adoption in the real world. Existing studies emphasized that the strong language priors of LVLMs can overpower visual information, causing hallucinations. However, the positive role of language priors is the key to a powerful LVLM. If the language priors are too weak, LVLMs will struggle to leverage rich parameter knowledge and instruction understanding abilities to complete tasks in challenging visual scenarios where visual information alone is insufficient. Therefore, we propose a benchmark called LanP to rethink the impact of Language Priors in LVLMs. It is designed to investigate how strong language priors are in current LVLMs. LanP consists of 170 images and 340 corresponding well-designed questions. Extensive experiments on 25 popular LVLMs reveal that many LVLMs' language priors are not strong enough to effectively aid question answering when objects are partially hidden. Many models, including GPT-4 Turbo, exhibit an accuracy below 0.5 in such a scenario.

LanP: Rethinking the Impact of Language Priors in Large Vision-Language Models

TL;DR

LanP introduces a targeted benchmark to reassess the positive role of language priors in LVLMs, emphasizing how priors can aid or hinder answers under visual ambiguity. Through 170 images and 340 carefully crafted yes/no questions across four categories, the study evaluates 25 LVLMs and finds that many models still rely insufficiently on language priors to resolve partially hidden objects. The results show a clear size-and-architecture dependency: larger LLMs and certain configurations can leverage priors to improve answers, yet Partially Hidden cases remain challenging, suggesting a need for balanced priors. Overall, LanP provides a framework for diagnosing and guiding the development of LVLMs that effectively fuse world knowledge with visual signals, with implications for reducing hallucination while maintaining reasoning capabilities.

Abstract

Large Vision-Language Models (LVLMs) have shown impressive performance in various tasks. However, LVLMs suffer from hallucination, which hinders their adoption in the real world. Existing studies emphasized that the strong language priors of LVLMs can overpower visual information, causing hallucinations. However, the positive role of language priors is the key to a powerful LVLM. If the language priors are too weak, LVLMs will struggle to leverage rich parameter knowledge and instruction understanding abilities to complete tasks in challenging visual scenarios where visual information alone is insufficient. Therefore, we propose a benchmark called LanP to rethink the impact of Language Priors in LVLMs. It is designed to investigate how strong language priors are in current LVLMs. LanP consists of 170 images and 340 corresponding well-designed questions. Extensive experiments on 25 popular LVLMs reveal that many LVLMs' language priors are not strong enough to effectively aid question answering when objects are partially hidden. Many models, including GPT-4 Turbo, exhibit an accuracy below 0.5 in such a scenario.

Paper Structure

This paper contains 17 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. The constructed benchmark - LanP
  4. Overview of LanP
  5. Dataset construction
  6. Image collection
  7. Question design
  8. Experiments
  9. Experimental Settings
  10. Experimental Results
  11. Overall Results
  12. Performance on Env, Blur, and Tiny Parts
  13. Performance on Partially Hidden Part
  14. Impact of Model Size
  15. Conclusion
  16. ...and 2 more sections

Figures (4)

  • Figure 1: An illustration of different roles of language priors in LVLMs. The left half of the image shows an example where language priors have a negative impact, while the right half shows an example where language priors bring a positive impact.
  • Figure 2: Some examples of LanP. There are four categories in our benchmark: Blur, Environment, Partially Hidden, and Tiny. We show one sample figure and corresponding questions for each category.
  • Figure 3: Image Statistics of Four Categories in the LanP Dataset.
  • Figure 4: Some questions that GPT-4 Turbo answers incorrectly.