Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VERA: Identifying and Leveraging Visual Evidence Retrieval Heads in Long-Context Understanding

Rongcan Pei, Huan Li, Fang Guo, Qi Zhu

TL;DR

The work identifies Visual Evidence Retrieval (VER) Heads as a sparse, dynamic subset of attention heads that ground long-context reasoning in visual documents. It defines a VER Score and shows VER heads are universal and causally impactful, with masking degrading performance and fine-tuning increasing their activation. Building on this, the authors introduce VERA, a training-free inference-time augmentation that detects uncertainty via entropy spikes, selects VER heads, retrieves critical patches, and verbalizes the visual evidence to improve reasoning across five benchmarks, achieving about $21.3\%$ and $20.1\%$ relative gains on Qwen3-VL-8B-Instruct and GLM-4.1V-Thinking respectively. The approach yields robust improvements across model architectures and unseen domains, highlighting a path toward interpretable, evidence-grounded long-context VLMs with efficient visual-text compression.

Abstract

While Vision-Language Models (VLMs) have shown promise in textual understanding, they face significant challenges when handling long context and complex reasoning tasks. In this paper, we dissect the internal mechanisms governing long-context processing in VLMs to understand their performance bottlenecks. Through the lens of attention analysis, we identify specific Visual Evidence Retrieval (VER) Heads - a sparse, dynamic set of attention heads critical for locating visual cues during reasoning, distinct from static OCR heads. We demonstrate that these heads are causal to model performance; masking them leads to significant degradation. Leveraging this discovery, we propose VERA (Visual Evidence Retrieval Augmentation), a training-free framework that detects model uncertainty (i.e., entropy) to trigger the explicit verbalization of visual evidence attended by VER heads. Comprehensive experiments demonstrate that VERA significantly improves long-context understanding of open-source VLMs: it yields an average relative improvement of 21.3% on Qwen3-VL-8B-Instruct and 20.1% on GLM-4.1V-Thinking across five benchmarks.

VERA: Identifying and Leveraging Visual Evidence Retrieval Heads in Long-Context Understanding

TL;DR

The work identifies Visual Evidence Retrieval (VER) Heads as a sparse, dynamic subset of attention heads that ground long-context reasoning in visual documents. It defines a VER Score and shows VER heads are universal and causally impactful, with masking degrading performance and fine-tuning increasing their activation. Building on this, the authors introduce VERA, a training-free inference-time augmentation that detects uncertainty via entropy spikes, selects VER heads, retrieves critical patches, and verbalizes the visual evidence to improve reasoning across five benchmarks, achieving about and relative gains on Qwen3-VL-8B-Instruct and GLM-4.1V-Thinking respectively. The approach yields robust improvements across model architectures and unseen domains, highlighting a path toward interpretable, evidence-grounded long-context VLMs with efficient visual-text compression.

Abstract

While Vision-Language Models (VLMs) have shown promise in textual understanding, they face significant challenges when handling long context and complex reasoning tasks. In this paper, we dissect the internal mechanisms governing long-context processing in VLMs to understand their performance bottlenecks. Through the lens of attention analysis, we identify specific Visual Evidence Retrieval (VER) Heads - a sparse, dynamic set of attention heads critical for locating visual cues during reasoning, distinct from static OCR heads. We demonstrate that these heads are causal to model performance; masking them leads to significant degradation. Leveraging this discovery, we propose VERA (Visual Evidence Retrieval Augmentation), a training-free framework that detects model uncertainty (i.e., entropy) to trigger the explicit verbalization of visual evidence attended by VER heads. Comprehensive experiments demonstrate that VERA significantly improves long-context understanding of open-source VLMs: it yields an average relative improvement of 21.3% on Qwen3-VL-8B-Instruct and 20.1% on GLM-4.1V-Thinking across five benchmarks.
Paper Structure (43 sections, 6 equations, 12 figures, 5 tables, 1 algorithm)

This paper contains 43 sections, 6 equations, 12 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visual Document Understanding.
  4. Mechanistic Interpretability and Attention Analysis.
  5. Long-context VLM Enhancement.
  6. Characterizing VER Heads
  7. Definition and Identification Method
  8. Text-to-Image Rendering.
  9. Visual Evidence Retrieval Score (VER Score).
  10. Quantitative Analysis of VER Heads
  11. (1) VER heads are sparse yet universal—they exist across different VLMs and remain consistent across datasets.
  12. (2) VER heads are functionally critical for long-context reasoning—masking them degrades performance, while masking other head types does not.
  13. (3) Does fine-tuning activate new VER heads, enhance existing ones, or both?
  14. Harnessing VER Heads in VLM Reasoning
  15. Identifying VER Heads for Different Models
  16. ...and 28 more sections

Figures (12)

  • Figure 1: Performance comparison between image and text inputs on the NIAH task (left) and long-context reasoning task (right).
  • Figure 2: Challenges in high-compression VLMs and illustration of visual retrieval augmentation.
  • Figure 3: Calculation of VER Score of a specific head.
  • Figure 4: Spearman's rank correlation coefficients of Visual Retrieval Score distributions across different datasets.
  • Figure 5: The distribution of Visual Retrieval Scores across attention heads.
  • ...and 7 more figures