Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding

TL;DR

The paper reveals that a small set of localization heads within frozen LVLMs inherently ground text-referred objects in images, enabling a training-free visual grounding framework. By evaluating two criteria—$S_{img}^{\ell,h}$ and $H({\bm{A}}^{\ell,h})$—the authors identify a handful of localization heads whose attention concentrates on relevant regions; aggregating their attention maps (with Gaussian smoothing) yields bounding boxes or masks without fine-tuning. Across ten LVLMs and standard REC/RES benchmarks, this approach outperforms other training-free methods and rivals fine-tuned methods, particularly as model size grows. The results underscore that LVLMs possess intrinsic text-vision grounding capabilities, offer interpretable attention-based explanations for grounding, and open avenues for real-world applications and image editing via mask guidance. $S_{img}^{\ell,h}$ and $H({\bm{A}}^{\ell,h})$ provide a principled mechanism to extract these localization heads, making the method robust and broadly applicable.

Abstract

Visual grounding seeks to localize the image region corresponding to a free-form text description. Recently, the strong multimodal capabilities of Large Vision-Language Models (LVLMs) have driven substantial improvements in visual grounding, though they inevitably require fine-tuning and additional model components to explicitly generate bounding boxes or segmentation masks. However, we discover that a few attention heads in frozen LVLMs demonstrate strong visual grounding capabilities. We refer to these heads, which consistently capture object locations related to text semantics, as localization heads. Using localization heads, we introduce a straightforward and effective training-free visual grounding framework that utilizes text-to-image attention maps from localization heads to identify the target objects. Surprisingly, only three out of thousands of attention heads are sufficient to achieve competitive localization performance compared to existing LVLM-based visual grounding methods that require fine-tuning. Our findings suggest that LVLMs can innately ground objects based on a deep comprehension of the text-image relationship, as they implicitly focus on relevant image regions to generate informative text outputs. All the source codes will be made available to the public.

Figures (19)

• Figure 1: Visualization of the text-to-image attention maps from LLaVA-1.5-7B LLaVA1.5. While the average attention map initially seems uninformative for localization, a closer examination reveals that LVLM possesses built-in localization heads that consistently capture key areas of an image corresponding to the referring text, regardless of sample variations. L14 H24 refers to the 24th attention head in the 14th layer of the LVLM.
• Figure 2: Comparison of LVLM frameworks for visual grounding. (a) Existing methods generally fine-tune a LVLM to leverage specialized tokens (e.g., $\mathtt{[SEG]}$) or language descriptions for visual grounding. (b) Our framework utilizes the attention maps of only a few localization heads from frozen LVLMs.
• Figure 3: Average $S_{\text{img}}^{\ell,h}$ values for each attention head. We sort the heads in ascending order of $S_{\text{img}}^{\ell,h}$. Attention heads with $S_{\text{img}}^{\ell,h} \geq \tau$ are considered to effectively attend to the image, where $\tau$ is the threshold determined by the maximum curvature in the graph.
• Figure 4: Illustration of the process for calculating spatial entropy. The attention map is binarized, and the spatial entropy is computed based on the sizes of its connected components $\{C_{i}\}_{i=1}^{N}$.
• Figure 5: Overview of finding localization heads. We first identify heads with high attention sum. Then, we evaluate spatial entropy for each head and select 10 heads with the lowest spatial entropy. We repeat this process for 1,000 image-text pairs and calculate the selection frequency of each head.