Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping

TL;DR

AttWarp tackles the challenge of fine-grained perceptual grounding in multimodal large language models by introducing a test-time, attention-guided image warping that reallocates spatial resolution to query-relevant regions while preserving global image structure. The method operates without modifying MLLM weights, deriving a rectilinear warp from cross-modal attention through marginal attention profiles and inverse CDF warping, and extends with AttWarp-Chain for iterative refinement and AttWarp-Distill for fast single-pass inference. Empirical results across five benchmarks and multiple backbones show consistent accuracy gains, improved compositional and spatial reasoning, and reduced hallucinations, with AttWarp-Distill offering significant speedups and AttWarp-Chain providing additional gains via adaptive termination. The findings demonstrate that input-level, information-preserving transformations can meaningfully enhance visual grounding in MLLMs and that AttWarp generalizes across external attention sources and newer architectures, offering a practical, plug-in tool for improving VQA and related tasks.

Abstract

Multimodal large language models (MLLMs) often miss small details and spatial relations in cluttered scenes, leading to errors in fine-grained perceptual grounding. We introduce AttWarp, a lightweight method that allocates more resolution to query-relevant content while compressing less informative areas, all while preserving global context. At test time, the approach uses an MLLM's cross-modal attention to perform rectilinear warping of the input image, reallocating spatial resolution toward regions the model deems important, without changing model weights or architecture. This attention-guided warping preserves all original image information but redistributes it non-uniformly, so small objects and subtle relationships become easier for the same model to read while the global layout remains intact. Across five benchmarks (TextVQA, GQA, DocVQA, POPE, MMMU) and four MLLMs (LLaVA, Qwen-VL, InternVL, and InstructBLIP), AttWarp consistently improves accuracy, strengthens compositional reasoning, and reduces hallucinations, outperforming four competitive baselines that manipulate raw images at test time. Together, these results show that attention-guided warping prioritizes information relevant to the query while preserving context, and that the same MLLMs perform better when given such warped inputs.

Figures (14)

• Figure 1: AttWarp overview. Given a query, our method extracts cross-modal attention maps from the MLLM's language decoder, aggregates them into marginal attention profiles, and uses rectilinear warping to expand high-attention regions while compressing low-attention areas. The warped image is then processed by the same MLLM, which now produces the correct answer.
• Figure 2: AttWarp-Distill architecture: CLIP vision tokens are FiLM‑modulated by text and projected to 1D marginal predictors.
• Figure 3: The figure shows the original image, the rectilinear warping function visualized as a warped grid, and the corresponding warped image. AttWarp enhances compositional and spatial reasoning, as illustrated by examples from the GQA dataset: (a) expanding the region containing the zebra behind the path, (b) expanding the television below the artwork, and (c) expanding the walls of the building. It also improves text understanding in document-based and real-world images: (d) expanding the phrase "consumer focus groups" (DocVQA), and (e) expanding the player's name on the board (TextVQA). Additionally, it aids fine-grained perception of small and occluded objects, exemplified by (f) the expansion of a bottle region in the POPE dataset.
• Figure 4: Qualitative comparison of AttWarp and baselines of test-time visual intervention. While plausible, the answers generated by prior works are incorrect. The answer (A) for each case is generated with LLaVA.
• Figure 5: AttWarp-Chain improves on AttWarp