CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models

TL;DR

CoF tackles the difficulty of fine-grained visual understanding in multimodal large language models by decomposing image comprehension into two stages: coarse-grained location grounding to identify an approximate target region, and fine-grained attention reweighting to emphasize details within that region. It operationalizes this by using a grounding prompt to produce region coordinates, followed by turning those coordinates into a binary mask to modulate the model's attention via $\hat{A} = \text{softmax}(\log(\lambda) \cdot M + A)$. Evaluations on LLaVA and InstructBLIP across benchmarks MME, MMBench, and POPE show improved grounding, better fine-grained understanding, and reduced hallucinations compared to baselines. The approach is interpretable and model-agnostic, with potential for broader application and further refinement of attention mechanisms in multimodal reasoning.

Abstract

The impressive performance of Large Language Model (LLM) has prompted researchers to develop Multi-modal LLM (MLLM), which has shown great potential for various multi-modal tasks. However, current MLLM often struggles to effectively address fine-grained multi-modal challenges. We argue that this limitation is closely linked to the models' visual grounding capabilities. The restricted spatial awareness and perceptual acuity of visual encoders frequently lead to interference from irrelevant background information in images, causing the models to overlook subtle but crucial details. As a result, achieving fine-grained regional visual comprehension becomes difficult. In this paper, we break down multi-modal understanding into two stages, from Coarse to Fine (CoF). In the first stage, we prompt the MLLM to locate the approximate area of the answer. In the second stage, we further enhance the model's focus on relevant areas within the image through visual prompt engineering, adjusting attention weights of pertinent regions. This, in turn, improves both visual grounding and overall performance in downstream tasks. Our experiments show that this approach significantly boosts the performance of baseline models, demonstrating notable generalization and effectiveness. Our CoF approach is available online at https://github.com/Gavin001201/CoF.

Figures (2)

• Figure 1: Overview of (a) baseline approach and (b) our CoF approach. CoF consists of two stages: (1) Location Grounding and (2) Attention Reweighting. In the first stage, the MLLM takes the question Q, the grounding prompt P$_\text{g}$, and image I as input and obtain the coarse-grained coordinates of the answer region. In the second stage, we reweight the attention score of the answer region according to the coarse-grained coordinates obtained in the first stage to guide the model to focus on the answer region.
• Figure 2: Overview of CoF approach. (a) In the first stage, the model determines the answer area in the input image based on the question. (b) Then the output coordinates are post-processed and converted into a binary mask matrix, and the attention map of the input image is reweighted according to the mask matrix.