VIB-Probe: Detecting and Mitigating Hallucinations in Vision-Language Models via Variational Information Bottleneck

TL;DR

This work tackles hallucinations in Vision-Language Models by introducing VIB-Probe, which extracts and compresses multi-layer, multi-head attention outputs using Variational Information Bottleneck to detect unfaithful generations. The detector produces token-level hallucination risk scores, enabling a training-free inference-time mitigation that suppresses the most influential attention heads via gradient-based attribution. Across four benchmarks and multiple LVLM architectures, VIB-Probe achieves state-of-the-art performance in both hallucination detection and mitigation, with robust transferability and resilience to image perturbations. The approach emphasizes turning internal model signals into actionable controls, offering a practical and scalable path to more faithful multimodal generation. The authors also provide extensive ablations demonstrating the critical role of the IB constraint and the benefit of multi-layer feature aggregation, suggesting strong generalization to diverse VLM settings.

Abstract

Vision-Language Models (VLMs) have demonstrated remarkable progress in multimodal tasks, but remain susceptible to hallucinations, where generated text deviates from the underlying visual content. Existing hallucination detection methods primarily rely on output logits or external verification tools, often overlooking their internal mechanisms. In this work, we investigate the outputs of internal attention heads, postulating that specific heads carry the primary signals for truthful generation.However, directly probing these high-dimensional states is challenging due to the entanglement of visual-linguistic syntax and noise. To address this, we propose VIB-Probe, a novel hallucination detection and mitigation framework leveraging the Variational Information Bottleneck (VIB) theory. Our method extracts discriminative patterns across layers and heads while filtering out semantic nuisances through the information bottleneck principle. Furthermore, by leveraging the gradients of our VIB probe, we identify attention heads with strong causal influence on hallucinations and introduce an inference-time intervention strategy for hallucination mitigation. Extensive experiments across diverse benchmarks demonstrate that VIB-Probe significantly outperforms existing baselines in both settings. Our code will be made publicly available.

Figures (3)

• Figure 1: Hallucination detection performance comparison across 6 benchmarks, based on the AUPRC metric. Our proposed VIB-Probe consistently achieves state-of-the-art overall results.
• Figure 2: Overview of the VIB-Probe framework. The Information Bottleneck (IB) theory is leveraged to detect and mitigate hallucinations by probing internal attention features. Stage 1: We extract raw output vectors from all attention heads across all Transformer layers ($L \times H$) during VLM decoding. Stage 2: The extracted features are fed into an IB Encoder, which compresses the high-dimensional input into a compact latent representation $z$. This process filters out task-irrelevant noise while retaining minimal sufficient statistics for prediction. Stage 3: Leveraging the trained VIB modules, inference-time mitigation is achieved by suppressing the attention heads with high risks of hallucination for each token, producing a more faithful output generation.
• Figure 3: Generalization gap from POPE-Popular to other test sets. A lower generalization gap indicates stronger transferability performance. Results are compared based on LLaVA-v1.5-7B.