Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs

TL;DR

The paper tackles LVLM hallucinations arising from dominance of language priors over visual evidence. It introduces Attention-space Contrastive Guidance (ACG), a training-free, single-pass mechanism that performs image-conditioned versus text-only attention contrast directly within self-attention layers, supplemented by a textual orthogonalization step to isolate visual contributions. Empirical results on CHAIR, POPE, and MMHal-Bench show state-of-the-art faithfulness and caption quality with substantial latency advantages over multi-pass methods. The approach demonstrates robust grounding improvements while maintaining reasonable caption fidelity, offering a practical, efficient alternative for real-time LVLM debiasing and paving the way for architecture-aware enhancements.

Abstract

Hallucinations in large vision-language models (LVLMs) often arise when language priors dominate over visual evidence, causing object misidentification and visually inconsistent descriptions. We address this issue by framing hallucination mitigation as contrastive guidance, steering generation toward visually grounded and semantically faithful text. This approach regulates the model's internal behavior by reducing over-dependence on language priors and contrasting visually grounded with language-only representations. We propose Attention-space Contrastive Guidance (ACG), a single-pass mechanism that operates within self-attention layers to construct both vision-language and language-only attention paths in a single forward computation. This integration enables computationally efficient guidance directly embedded in the model's representation contextualization. To correct approximation bias introduced by the single-pass formulation, we further apply an orthogonalized correction that removes components aligned with the language-only path, selectively amplifying visual contributions. Experiments on the CHAIR and POPE benchmarks show that ACG achieves state-of-the-art faithfulness and caption quality while significantly reducing computational cost. Our method establishes a principled and efficient alternative, reducing latency by up to 2x compared to prior contrastive decoding methods that require multiple forward passes.

Figures (7)

• Figure 1: Comparison of inference-time strategies for mitigating LVLM hallucinations. (a) Logit-level contrastive decoding, (b) hidden-state-level latent steering, (c) attention map intervention, and (d) the proposed Attention-space Contrastive Guidance (ACG).
• Figure 2: MMHal-Bench results on LLaVA-1.5. The radar chart reports GPT-4–judged hallucination scores across eight categories.
• Figure 3: Validation of the masked-unconditional approximation. (a) Increasing Gaussian noise (visual information loss) raises hallucination (CHAIR$_i$) and worsens fidelity (F1). (b) Mean text-to-image (T2I) attention ratio shows an overall downward trend as visual information is lost.
• Figure 4: Guidance–scale trade-off on LLaVA-1.5 (CHAIR, max 128). Increasing $\gamma$ lowers instance hallucination (CHAIR$_i$, blue; left axis) but eventually reduces object-level fidelity (F1, red; right axis). The dotted line marks the canonical setting $\gamma{=}2.4$.
• Figure 5: Qualitative Analysis. Comparison between responses generated by LLaVA-1.5 and LLaVA-1.5 with ACG(Ours). Hallucinated/Wrong and accurate content is highlighted in red and blue.