Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation

TL;DR

Adaptive Visual Reinforcement (AIR) is proposed, a training-free framework for MLLMs that substantially reduces hallucination while preserving general capabilities, establishing it as an effective solution for building reliable MLLMs.

Abstract

Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language reasoning, yet they remain vulnerable to hallucination, where generated content deviates from visual evidence. Existing mitigation strategies either require costly supervision during training or introduce additional latency at inference time. Recent vision enhancement methods attempt to address this issue by reinforcing visual tokens during decoding, but they typically inject all tokens indiscriminately, which causes interference from background regions and distracts the model from critical cues. To overcome this challenge, we propose Adaptive Visual Reinforcement (AIR), a training-free framework for MLLMs. AIR consists of two components. Prototype-based token reduction condenses the large pool of visual tokens into a compact subset to suppress redundancy. OT-guided patch reinforcement quantifies the alignment between hidden states and patch embeddings to selectively integrate the most consistent patches into feed-forward layers. As a result, AIR enhances the model's reliance on salient visual information and effectively mitigates hallucination. Extensive experiments across representative MLLMs demonstrate that AIR substantially reduces hallucination while preserving general capabilities, establishing it as an effective solution for building reliable MLLMs.

Figures (10)

• Figure 1: Analysis of existing hallucination mitigation strategies in multimodal large language models (MLLMs). (a) existing strategies hallucinate non-existent objects, while AIR produces faithful, image-grounded answers. (b) Similarity across decoder layers between hidden states and different visual tokens, showing that salient patches consistently achieve higher alignment than irrelevant ones. (c) Attention heatmaps comparing existing re-injection with AIR: prior methods spread attention to irrelevant regions, whereas AIR focuses on semantically critical areas. (d) AIR reduces hallucination across benchmarks with minimal impact on general multimodal performance.
• Figure 2: Overview of our proposed AIR framework. Given a multimodal input (image and question), visual features are extracted by the visual encoder and aligned with textual embeddings via the projector. Inside the Transformer layers, visual tokens are first compressed through prototype-based selection to remove redundancy, and then reinforced by patch-level alignment using optimal transport. These selective reinforcement strategies enrich the hidden states with salient visual cues, enabling AIR to produce safer, more faithful, and visually grounded responses.
• Figure 3: Performance under different numbers of retained visual tokens $\text{Top}_Q$.
• Figure 4: Results of GPT-4V-aided evaluation on LLaVA-Bench following the setting in VCD. Both metrics are on a scale of 10.
• Figure 5: Performance as the number of selected image patches increases.

Theorems & Definitions (2)

• Theorem 1
• proof