Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidance

TL;DR

The paper tackles hallucinations in Large Vision-Language Models caused by language priors, showing that hallucinated tokens can emerge during decoding and overshadow visually grounded tokens. It introduces Residual Decoding (ResDec), a training-free decoding strategy that uses historical information and $JSD$ to identify stable semantic phases and generate a residual logit stream, which is blended with the current logits via a weight $\alpha$. By performing confidence-weighted aggregation over a historical window and fusing history with present information through $p_{\text{ResDec}}(y_t|v,x,y_{<t})$, ResDec substantially reduces hallucinations while maintaining efficiency, achieving average improvements of $7.84\%$ in accuracy and $8.01\%$ in F1 across three LVLMs on POPE benchmarks and attaining state-of-the-art results on multiple hallucination and comprehensive benchmarks. This shows decoding-time interventions can significantly improve visual grounding and reliability in LVLMs with negligible overhead, offering a practical path toward more trustworthy multimodal generation.

Abstract

Large Vision-Language Models (LVLMs) can reason effectively from image-text inputs and perform well in various multimodal tasks. Despite this success, they are affected by language priors and often produce hallucinations. Hallucinations denote generated content that is grammatically and syntactically coherent, yet bears no match or direct relevance to actual visual input. To address this problem, we propose Residual Decoding (ResDec). It is a novel training-free method that uses historical information to aid decoding. The method relies on the internal implicit reasoning mechanism and token logits evolution mechanism of LVLMs to correct biases. Extensive experiments demonstrate that ResDec effectively suppresses hallucinations induced by language priors, significantly improves visual grounding, and reduces object hallucinations. In addition to mitigating hallucinations, ResDec also performs exceptionally well on comprehensive LVLM benchmarks, highlighting its broad applicability.

Figures (8)

• Figure 1: An overview of Residual Decoding. The figure illustrates the Language-Prior Hallucination by focusing on candidate tokens (i.e. A, B, C, and D), and indicates that Residual Decoding with residual guidance effectively mitigates such hallucinations.
• Figure 2: Accuracy of candidate answers at different time steps based on Qwen2.5-VL (7B) in MME and POPE. In both datasets, the candidate answers are binary Yes/No decisions.
• Figure 3: The trend of Jensen–Shannon Divergence (JSD) on $200$ randomly sampled POPE-MSCOCO examples using Qwen2.5-VL-Instruct (7B). JSD exhibits a distinct U-shape over time, which we divide into three phases: PSAP, SAP, and EDP.
• Figure 4: Ablation study on different sizes of Candidate Token Pool conducted on LLaVA-1.5 and focusing on POPE and MME. The optimal size of Candidate Token Pool ranges from $64$ to $512$, while excessively small or large sizes degrade performance by impairing SAP localization.
• Figure 5: Case study comparing regular decoding and residual decoding in image captioning. Hallucinated details in "regular" decoding outputs are highlighted in red.