Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steering

TL;DR

The paper tackles object hallucination in multimodal large language models by introducing Vision-Language Introspection (VLI), a training-free inference framework that combines Attributive Introspection for causal anchor localization with Interpretable Bi-Causal Steering for latent correction, plus Adaptive Confidence Calibration to curb blind confidence. The Bi-Causal Steering constructs a Counterfactual Anchor-Only and Context-Only states, deriving a steering delta Δ that is orthogonal to the linguistic prior z_lang and amplifies the visual signal, thereby increasing the latent signal to noise ratio. The approach is evaluated on MMHal-Bench and POPE across LLaVA-1.5 and Qwen3-VL, achieving state-of-the-art reductions in hallucination rates and improvements in POPE accuracy, with ablations confirming the necessity of the anchor and calibration components. The work also provides theoretical support for the mechanism, including a latent linear representation view and a formalization of the calibration as a regularizer against ungrounded certainty, offering a practical path toward more trustworthy multimodal reasoning without retraining.

Abstract

Object hallucination critically undermines the reliability of Multimodal Large Language Models, often stemming from a fundamental failure in cognitive introspection, where models blindly trust linguistic priors over specific visual evidence. Existing mitigations remain limited: contrastive decoding approaches operate superficially without rectifying internal semantic misalignments, while current latent steering methods rely on static vectors that lack instance-specific precision. We introduce Vision-Language Introspection (VLI), a training-free inference framework that simulates a metacognitive self-correction process. VLI first performs Attributive Introspection to diagnose hallucination risks via probabilistic conflict detection and localize the causal visual anchors. It then employs Interpretable Bi-Causal Steering to actively modulate the inference process, dynamically isolating visual evidence from background noise while neutralizing blind confidence through adaptive calibration. VLI achieves state-of-the-art performance on advanced models, reducing object hallucination rates by 12.67% on MMHal-Bench and improving accuracy by 5.8% on POPE.

Figures (13)

• Figure 1: Overview of VLI framework. VLI first detects Introspective Conflict (§\ref{['sec:suspect_token']}) between grounded ($h_g$) and ungrounded ($h_u$) paths to localize the causal anchor $\mathcal{M}_s$ (§\ref{['sec:anchor_extraction']}) via purified expert attention (§\ref{['sec:attention_purification']}, Fig. \ref{['fig:heatmap']}). It then applies Bi-Causal Steering (§\ref{['sec:intervention']}) using the robust difference vector ($h_a - h_c$), which counters the scope instability of individual counterfactual states (Fig. \ref{['fig:Layer-wise analysis of hidden state shifts']}). Finally, Adaptive Confidence Calibration (§\ref{['sec:calibration']}) penalizes blind confidence to mitigate persistent hallucinations.
• Figure 2: Detailed performance of different models on the eight categories in MMHAL-BENCH, where “Overall” indicates the averaged performance across all categories. A higher score indicates that the generated response contains fewer hallucinations and more information.
• Figure 3: Heatmap of max-pooled attention intensity across layers and heads.
• Figure 4: Layer-wise analysis of hidden state shifts. (a) Presenting all tokens in a single representative sample from MMHal-Bench. (b) Focusing on introspected tokens across the MMHal-Bench dataset. The solid lines denote the median JS divergence, while the shaded regions indicate the interquartile range (IQR).
• Figure 5: Hyperparameter sensitivity analysis on LLaVA-1.5 and Qwen3-VL. The solid lines represent the average performance of VLI, while the shaded regions indicate the standard deviation, highlighting the stability of our method. The dashed lines denote the baseline performance of the original models (Origin) without intervention.