HalluShift++: Bridging Language and Vision through Internal Representation Shifts for Hierarchical Hallucinations in MLLMs

TL;DR

HalluShift++ extends internal-representation analysis from text-only LLMs to multimodal LLMs to detect and classify visual-grounding hallucinations. It introduces 74 features (including 12 new multimodal-specific ones), semantic chunking into object/attribute/relation units, and a hierarchical ground-truth-matching scheme with a multi-class membership function. Evaluated on MS-COCO and LLaVA across eight MLLMs and LLM QA tasks, it achieves substantial AUC-ROC improvements over HalluShift, including scale-agnostic gains on small models. The approach reveals that cross-layer inconsistency, attention dispersion, and confidence degradation are robust hallmarks of multimodal hallucination, enabling practical, fine-grained detection and analysis.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in vision-language understanding tasks. While these models often produce linguistically coherent output, they often suffer from hallucinations, generating descriptions that are factually inconsistent with the visual content, potentially leading to adverse consequences. Therefore, the assessment of hallucinations in MLLM has become increasingly crucial in the model development process. Contemporary methodologies predominantly depend on external LLM evaluators, which are themselves susceptible to hallucinations and may present challenges in terms of domain adaptation. In this study, we propose the hypothesis that hallucination manifests as measurable irregularities within the internal layer dynamics of MLLMs, not merely due to distributional shifts but also in the context of layer-wise analysis of specific assumptions. By incorporating such modifications, \textsc{\textsc{HalluShift++}} broadens the efficacy of hallucination detection from text-based large language models (LLMs) to encompass multimodal scenarios. Our codebase is available at https://github.com/C0mRD/HalluShift_Plus.

Figures (3)

• Figure 1: HalluShift++ in action: Left: MLLMs generate descriptions for visual inputs that may contain hallucinations. Right: Our framework provides hierarchical hallucination scores across three types (Category, Attribute, Relation).
• Figure 2: Illustration of our proposed HalluShift++ framework. First, we extract features from the internal states of MLLMs including: (1) original HalluShift features (1-62), and (2) novel HalluShift++ features (63-74, highlighted in blue boxes) comprising 2 attention concentration features (ACF) measuring focus dispersion via Gini coefficients, 2 layer consistency features (LCF) detecting cross-layer disagreement, and perplexity/confidence features capturing uncertainty patterns. Color transitions from green to red in internal distributions indicate hallucination-correlated patterns. Second, semantic chunking decomposes generated text into object, attribute, and relation components. Finally, our membership function processes the combined 74-dimensional feature vectors to assign hierarchical hallucination scores across four categories (Correctness, Attribute, Category, Relation) rather than binary classification.
• Figure 3: Feature importance analysis showing top 10 features for the best performing model (Llama-3.2-11B-Vision). The horizontal bar chart shows the contribution of each feature to hallucination detection accuracy, with semantic context features dominating the rankings while HalluShift++ features provide significant improvements. Feature names are interpretable rather than numerical for clarity.