Attention Prompting on Image for Large Vision-Language Models

TL;DR

Attention Prompting on Image (API) introduces a text-query-guided heatmap overlay on input images to steer LVLM perception, generating attribution heatmaps with an auxiliary LVLM (e.g., CLIP or LLaVA) and converting them into a pixel-space overlay that is fed to the LVLM without fine-tuning. The approach enables explicit query-driven visual guidance and supports ensembling and Self-Reflection in LVLMs, using heatmaps derived from either CLIP or LLaVA attention patterns and fused through mean filtering. Across six VL datasets and multiple LVLMs, API yields consistent performance gains and demonstrates robustness to hallucination, while ablations reveal the importance of auxiliary-model scale, heatmap smoothing, and layer choice. The work highlights the value of query-aware visual prompting for LVLMs and provides practical insights into attribution-map extraction, fusion, and deployment considerations for improved vision-language reasoning.

Abstract

Compared with Large Language Models (LLMs), Large Vision-Language Models (LVLMs) can also accept images as input, thus showcasing more interesting emergent capabilities and demonstrating impressive performance on various vision-language tasks. Motivated by text prompting in LLMs, visual prompting has been explored to enhance LVLMs' capabilities of perceiving visual information. However, previous visual prompting techniques solely process visual inputs without considering text queries, limiting the models' ability to follow text instructions to complete tasks. To fill this gap, in this work, we propose a new prompting technique named Attention Prompting on Image, which just simply overlays a text-query-guided attention heatmap on the original input image and effectively enhances LVLM on various tasks. Specifically, we generate an attention heatmap for the input image dependent on the text query with an auxiliary model like CLIP. Then the heatmap simply multiplies the pixel values of the original image to obtain the actual input image for the LVLM. Extensive experiments on various vison-language benchmarks verify the effectiveness of our technique. For example, Attention Prompting on Image improves LLaVA-1.5 by 3.8% and 2.9% on MM-Vet and LLaVA-Wild benchmarks, respectively.

Figures (12)

• Figure 1: Comparison of the proposed Attention Prompting on Image ($\mathcal{API}$) with the naive VQA.$\mathcal{API}$ provides hints for LVLM by simply overlying a heatmap on the image.
• Figure 2: In complex images including multiple objects, our method accurately highlights the fruits and masks the other objects, thereby simplifying the scene and facilitating the LVLM's inference of spatial relationships.
• Figure 3: Our method identifies regions related to the objects, thereby assisting the LVLM in spatial reasoning.
• Figure 4: Our method assists LVLM's recognition process by highlighting the corresponding steps in the flowchart.
• Figure 5: In this example, our method enhances LVLM's OCR capability by masking background areas and highlighting the regions that require OCR.