Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation

TL;DR

Vision-language models still hallucinate due to over-reliance on linguistic priors and misallocated visual attention. GIFT introduces a gaze-shift-guided visual saliency map computed by tracking positive changes in visual attention to information-rich query tokens, then uses this map to jointly amplify visual and query attention at select decoding layers to improve cross-modal fusion. The approach mitigates visual attention sink, preserves reasoning capabilities, and yields up to 20.7% gains on CHAIR, 15.9% on MMHal-Bench, and 3.0% on POPE across multiple models with low overhead. Overall, GIFT offers a lightweight, inference-time solution that enhances hallucination grounding while maintaining general vision-language performance.

Abstract

Vision language models (VLMs) often generate hallucination, i.e., content that cannot be substantiated by either textual or visual inputs. Prior work primarily attributes this to over-reliance on linguistic prior knowledge rather than visual inputs. Some methods attempt to mitigate hallucination by amplifying visual token attention proportionally to their attention scores. However, these methods overlook the visual attention sink problem, where attention is frequently misallocated to task-irrelevant visual regions, and neglect cross-modal fusion balance by enhancing only visual attention without adjusting attention to the user query. This can result in amplifying incorrect areas while failing to properly interpret the user query. To address these challenges, we propose a simple yet effective method called Gaze Shift-Guided Cross-modal Fusion Enhancement (GIFT). GIFT pre-computes a holistic visual saliency map by tracking positive changes in visual attention, or "gaze shifts", during user query comprehension, and leverages this map to amplify attention to both salient visual information and the user query at each decoding step. This reduces the impact of visual attention sink, as irrelevant tokens exhibit minimal shifts, while ensuring balanced cross-modal fusion for well-integrated representation. Extensive experiments show that GIFT effectively mitigates hallucination in VLMs across both generative and classification tasks, achieving up to 20.7% improvement over greedy decoding, while maintaining general vision-language performance with low computational overhead.

Figures (12)

• Figure 1: Examples of visual saliency maps from LLaVA 1.5 7B. The vanilla method (Static) averages visual token attention over all query tokens, while the proposed method (Shift) averages positive changes in visual token attention over information-rich query tokens. Shift more effectively highlights task-relevant visual regions and mitigates the impact of visual attention sink, where irrelevant regions receive erroneously high saliency scores.
• Figure 2: Volume of visual attention positive shifts in VLMs when processing information-rich query tokens across layers. The volume reflects how strongly the model reallocates focus within visual regions, with the largest shifts occurring in early to middle layers, indicating that VLMs settle on relevant visual regions once identified.
• Figure 3: Illustration of GIFT. GIFT tracks positive changes in visual attention, i.e., "gaze shifts" across information-rich query tokens to compute a visual saliency map, which guides enhancement of salient visual attention while preserving cross-modal fusion balance.
• Figure 4: Attention proportions of query and visual tokens from output tokens across layers. The attention proportions of query and visual tokens follow similar patterns across layers, suggesting that effective cross-modal fusion relies on contributions from both modalities.
• Figure 5: Performance of LLaVA-1.5 7B on the POPE and MME datasets with varying enhancement coefficients $\alpha$.