HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Models

TL;DR

HalluRNN tackles hallucinations in Large Vision-Language Models by introducing a cross-layer recurrent mechanism. It implements a Dual-Gated Depth Propagation Unit (DG-DPU) shared across Transformer layers, enabling adaptive information propagation and explicit modeling of inter-layer state evolution. The DG-DPU employs a Constraint Gate and a Correction Gate to control coarse-to-fine updates, and only the DG-DPU is fine-tuned for efficiency. Empirical results across POPE, MM-Vet, CHAIR, and MMMU benchmarks show HalluRNN consistently reduces hallucinations and transfers across models, highlighting its robustness and broad applicability.

Abstract

Though Large Vision-Language Models (LVLMs) have achieved remarkable performance across various tasks, they are still prone to hallucinations-generating outputs that are textually plausible but visually ungrounded. While prior approaches generally address this issue through data-centric fine-tuning or innovative decoding strategies, these methods often require substantial resources or task-specific configurations. In this work, we introduce an architecture-level solution, HalluRNN, which enhances model stability through recurrent cross-layer reasoning. Specifically, we propose a novel Dual-Gated Depth Propagation Unit (DG-DPU) module, which is shared across layers and recurrently refines hidden states. This allows for the adaptive propagation of information throughout the model, enforces consistency across layers, and mitigates hallucinations caused by representational drift. By fine-tuning only the DG-DPU module, HalluRNN achieves strong and robust performance across multiple benchmarks.

Figures (12)

• Figure 1: The performance of DAMO and DeCO under different hyperparameter settings is compared on two benchmarks using LLaVA-1.5, with the Vanilla setting results also provided for reference. (a) Comparison on MME benchmark. The red and blue bars represent DAMO with the MME setting and POPE setting, respectively, along with DeCO under different early exit configurations. (b) Comparison on POPE benchmark. Subfigure $i$ displays DAMO's performance for accuracy (left) and F1 score (right). Subfigure $ii$ illustrates DeCO's performance for accuracy (left) and F1 score (right).
• Figure 2: (a) Vanilla decoding process of an $N$-layer LVLM. (b) Proposed HalluRNN architecture, featuring a Dual-Gate Dynamic Propagation Unit (DG-DPU) that enforces layer-wise consistency via shared gating weights.
• Figure 3: (a) The GRU block. (b) The proposed DG-DPU block. The symbols $\oplus$, $\circleddash$ and $\copyright$ represent matrix addition, subtraction and concatenation, respectively. Specifically, $\odot$ denotes Hadamard product, while $\otimes$ represents scalar multiplication.
• Figure 4: Vanilla
• Figure 5: HalluRNN