Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models

Kai Tang, Jinhao You, Xiuqi Ge, Hanze Li, Yichen Guo, Xiande Huang

TL;DR

This work tackles hallucinations in Large Vision-Language Models (LVLMs) by proposing Decoding with Inter-layer Consistency via Layer Aggregation (DCLA), a training-free decoding strategy that enforces semantic stability across transformer layers. DCLA builds a dynamic inter-layer semantic reference by aggregating representations from earlier layers and refines later layers only when semantic drift is detected, via a parameterized fusion with $\alpha$ and an adaptive trigger $\tau$. The approach uses a cosine-similarity based criterion to selectively correct layers, enabling robust performance without retraining. Comprehensive experiments on MME, POPE, VizWiz, and MM-Vet across four LVLMs show reduced hallucinations and improved reliability, outperforming prior decoding baselines in both standard and adversarial settings, with strong generalization and practicality.

Abstract

Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucinations-generating content that is inconsistent with the input image. Existing training-free hallucination mitigation methods often suffer from unstable performance and high sensitivity to hyperparameter settings, limiting their practicality and broader adoption. In this paper, we propose a novel decoding mechanism, Decoding with Inter-layer Consistency via Layer Aggregation (DCLA), which requires no retraining, fine-tuning, or access to external knowledge bases. Specifically, our approach constructs a dynamic semantic reference by aggregating representations from previous layers, and corrects semantically deviated layers to enforce inter-layer consistency. The method allows DCLA to robustly mitigate hallucinations across multiple LVLMs. Experiments on hallucination benchmarks such as MME and POPE demonstrate that DCLA effectively reduces hallucinations while enhancing the reliability and performance of LVLMs.

Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models

TL;DR

This work tackles hallucinations in Large Vision-Language Models (LVLMs) by proposing Decoding with Inter-layer Consistency via Layer Aggregation (DCLA), a training-free decoding strategy that enforces semantic stability across transformer layers. DCLA builds a dynamic inter-layer semantic reference by aggregating representations from earlier layers and refines later layers only when semantic drift is detected, via a parameterized fusion with and an adaptive trigger . The approach uses a cosine-similarity based criterion to selectively correct layers, enabling robust performance without retraining. Comprehensive experiments on MME, POPE, VizWiz, and MM-Vet across four LVLMs show reduced hallucinations and improved reliability, outperforming prior decoding baselines in both standard and adversarial settings, with strong generalization and practicality.

Abstract

Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucinations-generating content that is inconsistent with the input image. Existing training-free hallucination mitigation methods often suffer from unstable performance and high sensitivity to hyperparameter settings, limiting their practicality and broader adoption. In this paper, we propose a novel decoding mechanism, Decoding with Inter-layer Consistency via Layer Aggregation (DCLA), which requires no retraining, fine-tuning, or access to external knowledge bases. Specifically, our approach constructs a dynamic semantic reference by aggregating representations from previous layers, and corrects semantically deviated layers to enforce inter-layer consistency. The method allows DCLA to robustly mitigate hallucinations across multiple LVLMs. Experiments on hallucination benchmarks such as MME and POPE demonstrate that DCLA effectively reduces hallucinations while enhancing the reliability and performance of LVLMs.
Paper Structure (28 sections, 6 equations, 4 figures, 10 tables)

This paper contains 28 sections, 6 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Vision-Language Models
  4. Layer Aggregation Mechanisms
  5. Hallucinations in LVLMs
  6. Method
  7. Decoding in Large Vision-Language Models
  8. Motivation
  9. Decoding with Inter-Layer Consistency via Layer Aggregation
  10. Layer Aggregation
  11. Decoding with Inter-Layer Consistency
  12. Adaptive Layer Aggregation Correction
  13. Experiment
  14. Setup
  15. Datasets
  16. ...and 13 more sections

Figures (4)

  • Figure 1: Illustration of Decoding with Inter‑Layer Consistency via Layer Aggregation (DCLA) in LLaVA1.5-7b. Vanilla decoding process produces hallucinations and thus generates incorrect answers. By aggregating hidden states across layers to refine the representation, DCLA effectively suppresses hallucinations and restores the ground‑truth answer.
  • Figure 2: Layer Aggregation mechanism to form a stable semantic reference.
  • Figure 3: (a) Comparison of different fixed refinement layers in DCLA (0–20th, 0–24th, 0–28th, 0–32nd) against standard decoding and our adaptive correction mechanism. (b)Evaluation of the dynamic correction mechanism and fixed correction on LLaVA1.5-7b, LLaVA-Next, LLaVA1.5-13b, and mPLUG-Owl2.
  • Figure 4: (a) Parameter sensitivity analysis of correction strength and trigger threshold on POPE subsets (Random, Popular, Adversarial). (b) Parameter sensitivity analysis of correction strength and trigger threshold on MME dataset.