Beyond Static Cropping: Layer-Adaptive Visual Localization and Decoding Enhancement

TL;DR

This work tackles the fixed visual token bottleneck in large vision-language models by showing that visual grounding is layer-dependent and query-sensitive. It introduces Visual Activation by Query (VAQ) to identify the most informative layer for a given query, and Visual Activation of Tokens (VAT) to guide decoding toward visually supported tokens. Built on these, LASER provides a training-free, layer-adaptive inference framework that performs query-aware localization via Con-ViCrop and counterfactual decoding to suppress language priors. Across multiple LVLMs and benchmarks, LASER consistently improves visual grounding and VQA accuracy, demonstrating the practical value of dynamic, query-conditioned grounding in complex multimodal reasoning.

Abstract

Large Vision-Language Models (LVLMs) have advanced rapidly by aligning visual patches with the text embedding space, but a fixed visual-token budget forces images to be resized to a uniform pretraining resolution, often erasing fine-grained details and causing hallucinations via over-reliance on language priors. Recent attention-guided enhancement (e.g., cropping or region-focused attention allocation) alleviates this, yet it commonly hinges on a static "magic layer" empirically chosen on simple recognition benchmarks and thus may not transfer to complex reasoning tasks. In contrast to this static assumption, we propose a dynamic perspective on visual grounding. Through a layer-wise sensitivity analysis, we demonstrate that visual grounding is a dynamic process: while simple object recognition tasks rely on middle layers, complex visual search and reasoning tasks require visual information to be reactivated at deeper layers. Based on this observation, we introduce Visual Activation by Query (VAQ), a metric that identifies the layer whose attention map is most relevant to query-specific visual grounding by measuring attention sensitivity to the input query. Building on VAQ, we further propose LASER (Layer-adaptive Attention-guided Selective visual and decoding Enhancement for Reasoning), a training-free inference procedure that adaptively selects task-appropriate layers for visual localization and question answering. Experiments across diverse VQA benchmarks show that LASER significantly improves VQA accuracy across tasks with varying levels of complexity.

Figures (5)

• Figure 1: Visualization of LVLM attention maps across varying layers and query complexities. Red dashed circles mark the regions with the highest attention. Top: For a simple query, the model attends to the correct visual evidence effectively at a middle layer. Bottom: For a complex query requiring multi-step reasoning, the model initially focuses on visually salient but query-irrelevant regions and converges on the correct target at a deeper layer. Our work dynamically finds the correct layer to zoom in to visual evidence.
• Figure 2: Illustration of the visual grounding mechanism. (1) Image Tokenization: the input image is encoded into visual tokens while explicitly preserving its $m \times n$ spatial grid structure. (2) Attention Mapping: attention weights assigned by the Language Model are projected back from the token sequence onto the 2D image plane to localize the grounded region with high attention from tokens.
• Figure 3: Overview of the LASER framework. LASER operates in three stages to improve visual grounding and enhance image comprehension for LVLMs. (1) Layer Selection: for each layer, we compute query-conditioned contrastive attention by subtracting visual attention without the query from that with the query, yielding a layer-wise VAQ score that measures how strongly visual attention is modulated by the query; the most visually activated layer $\ell^*$ is selected dynamically per instance. (2) VAQ-guided Localization: the contrastive attention map at $\ell^*$ is used to localize query-relevant regions and perform constrained visual cropping, producing a focused image input that strengthens visual evidence. (3) Decoding: counterfactual verification is enabled by masking the most query-relevant visual evidence to form a negative view; logits from positive and counterfactual streams are contrasted to compute VAT and guide contrastive decoding, while easier cases decode directly from the cropped image. This design adaptively aligns visual evidence selection, localization, and decoding with query difficulty.
• Figure 4: Layer-wise VAQ scores across queries on simple task of object existence detection (POPE benchmark) and complex task of visual reasoning (AOK-VQA benchmark). Bars show the VAQ scores at each layer averaged across different queries. Simple queries activate peaked visual attention at 14th layer while complex queries activate much more visual attention at later layers.
• Figure 5: The distribution of the most visually activated layers selected by VAQ across different queries in three benchmarks. The 14th layer is selected overwhelmingly in POPE while later layers are more frequently selected in other two benchmarks.