Head-Aware Visual Cropping: Enhancing Fine-Grained VQA with Attention-Guided Subimage

TL;DR

This paper tackles the challenge of fine-grained reasoning in multimodal large language models by addressing noisy attention aggregation through a training-free Head Aware Visual Cropping (HAVC) framework. HAVC first identifies expert visual heads via an OCR-guided diagnostic task, retaining only heads with genuine grounding ability. At inference, it refines these heads using spatial entropy and gradient sensitivity, fusing their signals into a Visual Cropping Guidance Map to crop a task-relevant subimage for the MLLM. The cropped subimage, paired with the original image and question, improves visual grounding and accuracy across multiple fine-grained VQA benchmarks, outperforming state-of-the-art cropping strategies and showing strong robustness across backbones.

Abstract

Multimodal Large Language Models (MLLMs) show strong performance in Visual Question Answering (VQA) but remain limited in fine-grained reasoning due to low-resolution inputs and noisy attention aggregation. We propose \textbf{Head Aware Visual Cropping (HAVC)}, a training-free method that improves visual grounding by leveraging a selectively refined subset of attention heads. HAVC first filters heads through an OCR-based diagnostic task, ensuring that only those with genuine grounding ability are retained. At inference, these heads are further refined using spatial entropy for stronger spatial concentration and gradient sensitivity for predictive contribution. The fused signals produce a reliable Visual Cropping Guidance Map, which highlights the most task-relevant region and guides the cropping of a subimage subsequently provided to the MLLM together with the image-question pair. Extensive experiments on multiple fine-grained VQA benchmarks demonstrate that HAVC consistently outperforms state-of-the-art cropping strategies, achieving more precise localization, stronger visual grounding, providing a simple yet effective strategy for enhancing precision in MLLMs.

Figures (4)

• Figure 1: Overview of identifying Expert Visual Heads. Through an OCR-based diagnostic task, the model generates all attention maps, and heads with a visual grounding score above 0.5 are retained as expert visual heads for further refinement.
• Figure 2: Overview of the Visual Cropping Guidance Map Generation Process. Expert visual heads are further refined by spatial entropy and fusion score filtering at inference time. Their attention maps are then aggregated to produce a guidance map, which highlights task-relevant regions and directs the final visual grounding.
• Figure 3: Sensitivity analysis of key parameters in our HAVC framework, evaluated on the TextVQA dataset. (a) Effect of score threshold for expert head filtering. (b) Effect of weighting factor $\alpha$ for balancing spatial entropy and gradient sensitivity. (c) Effect of the number of selected heads K. (d) Effect of temperature parameter $\tau$ in the softmax weighting.
• Figure 4: Qualitative comparison for the question "What is the color of the scarf?". Our method correctly localizes the target and provides the right answer, while the baselines (Original, ViCrop) fail.