ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Models

TL;DR

ClearSight identifies object hallucination in multimodal LLMs as a consequence of insufficient visual signal during middle-layer fusion ($8<l<15$) and shows that contrastive decoding, while ground truthing outputs to vision, can harm coherence and slow inference. It introduces Visual Amplification Fusion (VAF), a training-free plug-in that increases visual attention in mid-layer fusion via an attention-modulation rule with enhancement and suppression coefficients $\alpha$ and $\beta$, plus selective visual-perception head restriction. Empirically, VAF delivers notable hallucination reduction across POPE and MME benchmarks with approximately 3% and 7% improvements respectively, while preserving coherence and introducing negligible inference-time overhead relative to VCD/ICD. The work demonstrates a practical, scalable approach to grounding multimodal outputs in visual inputs, enabling safer deployment of vision-language systems in high-stakes settings.

Abstract

Contrastive decoding strategies are widely used to mitigate object hallucinations in multimodal large language models (MLLMs). By reducing over-reliance on language priors, these strategies ensure that generated content remains closely grounded in visual inputs, producing contextually accurate outputs. Since contrastive decoding requires no additional training or external tools, it offers both computational efficiency and versatility, making it highly attractive. However, these methods present two main limitations: (1) bluntly suppressing language priors can compromise coherence and accuracy of generated content, and (2) processing contrastive inputs adds computational load, significantly slowing inference speed. To address these challenges, we propose Visual Amplification Fusion (VAF), a plug-and-play technique that enhances attention to visual signals within the model's middle layers, where modality fusion predominantly occurs. This approach enables more effective capture of visual features, reducing the model's bias toward language modality. Experimental results demonstrate that VAF significantly reduces hallucinations across various MLLMs without affecting inference speed, while maintaining coherence and accuracy in generated outputs.

Figures (18)

• Figure 1: Illustration of Visual Contrastive Decoding. The hallucinated object "Teacher" is suppressed by contrasting with an output distribution prone to hallucinations. This method has two main drawbacks: (1) additional processing of distorted visual inputs greatly increases inference time; (2) subtracting the language prior disrupts content coherence.
• Figure 2: Impact of VCD on Model Performance. CIDEr scores are reported on the Nocaps benchmark, while Accuracy is presented for the ScienceQA benchmark. The use of VCD leads to a significant decline in model performance.
• Figure 3: The importance of intra-visual flow and visual-textual flow across various layers. The visual-textual information flow in the middle layers has a significant impact on prediction outcomes.
• Figure 4: Attention Distribution of Modal Information Across Model Layers. In the middle layers, the model allocates insufficient attention to visual features while disproportionately focusing on system prompts.
• Figure 5: Illustration of the Visual Amplification Fusion Method. In the middle layers, we select attention heads highly responsive to visual information, amplifying their focus on visual features while reducing unnecessary attention to system prompts.