Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Visual Dependency in Long-Context Reasoning for Large Vision-Language Models

Yucheng Zhou, Zhi Rao, Jun Wan, Jianbing Shen

TL;DR

The paper tackles the decline of large vision-language models in long-context reasoning due to diminished visual reliance. It introduces a training-free context-pruning method that removes low-importance textual tokens based on multi-head attention, preserving visuals to foster stronger visual dependency. The authors provide theoretical guarantees showing pruning concentrates attention and shifts reliance toward visual inputs, and validate the approach with comprehensive experiments on a SVIT-derived long-context dataset and the MMDU benchmark, achieving consistent gains and reduced inference time. The work highlights the layerwise dynamics of cross-modal attention, demonstrates robustness across model sizes, and suggests scaling laws between pruning rate and context length, offering a practical route to more reliable long-context LVLMs.

Abstract

Large Vision-Language Models (LVLMs) excel in cross-model tasks but experience performance declines in long-context reasoning due to overreliance on textual information and reduced visual dependency. In this study, we empirically analyze LVLMs in long-context reasoning, revealing that increased context length leads to a higher dependence on language at the expense of visual dependency. To address this issue, we propose a novel training-free context pruning method that selectively removes less critical textual information. Our approach enhances visual dependency and reduces textual noise, thereby improving LVLM performance in long-context reasoning. We validate our method by constructing a long-context dataset, demonstrating its effectiveness across various LVLMs. Moreover, further analysis confirms the robustness of different token pruning strategies and preliminary explores scaling laws between pruning rates and context length.

Rethinking Visual Dependency in Long-Context Reasoning for Large Vision-Language Models

TL;DR

The paper tackles the decline of large vision-language models in long-context reasoning due to diminished visual reliance. It introduces a training-free context-pruning method that removes low-importance textual tokens based on multi-head attention, preserving visuals to foster stronger visual dependency. The authors provide theoretical guarantees showing pruning concentrates attention and shifts reliance toward visual inputs, and validate the approach with comprehensive experiments on a SVIT-derived long-context dataset and the MMDU benchmark, achieving consistent gains and reduced inference time. The work highlights the layerwise dynamics of cross-modal attention, demonstrates robustness across model sizes, and suggests scaling laws between pruning rate and context length, offering a practical route to more reliable long-context LVLMs.

Abstract

Large Vision-Language Models (LVLMs) excel in cross-model tasks but experience performance declines in long-context reasoning due to overreliance on textual information and reduced visual dependency. In this study, we empirically analyze LVLMs in long-context reasoning, revealing that increased context length leads to a higher dependence on language at the expense of visual dependency. To address this issue, we propose a novel training-free context pruning method that selectively removes less critical textual information. Our approach enhances visual dependency and reduces textual noise, thereby improving LVLM performance in long-context reasoning. We validate our method by constructing a long-context dataset, demonstrating its effectiveness across various LVLMs. Moreover, further analysis confirms the robustness of different token pruning strategies and preliminary explores scaling laws between pruning rates and context length.

Paper Structure

This paper contains 41 sections, 18 equations, 18 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background and Notation
  3. Related Work (Summarized).
  4. Analysis on Visual Dependency of LVLMs in Long-Context Reasoning
  5. Experimental Setting.
  6. Does Context Length Affect Visual Dependency?
  7. Impact of Language Priors.
  8. Consistency Between Vision and Language Priors
  9. How Does Visual Content Affect Visual Dependency?
  10. Attention Weight Distribution Across Vision and Language Information
  11. Context Pruning to Improve Visual Dependency
  12. Context Pruning
  13. Theoretical Demonstration
  14. Proposition 1:
  15. Proof:
  16. ...and 26 more sections

Figures (18)

  • Figure 1: Performance of LVLMs on long-context reasoning.
  • Figure 2: Performance of LVLMs on text-only long contexts.
  • Figure 3: Consistency between vision and language priors.
  • Figure 4: The relationship between the proportion of target visual content in the image and performance.
  • Figure 5: The distribution of attention weights between different layers of LVLMs.
  • ...and 13 more figures