Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding

TL;DR

This work tackles object hallucination in large vision-language models by introducing Internal Fact-based Contrastive Decoding (IFCD), an inference-time, model-agnostic method that leverages internal representation editing to generate a pair of distributions ($P^+$ and $P^-$) and contrastively decode to suppress hallucinatory logits. A TruthX-based editing component reveals truthfulness in latent representations, and adaptive plausibility constraints limit the influence of dubious tokens. Evaluated on LLaVA 1.5 and InstructBLIP across POPE, MME, and MSCOCO, IFCD achieves consistent reductions in hallucinations and improvements in perceptual tasks, e.g., up to 9% POPE accuracy and 8% MME improvements, while preserving text generation quality. The results indicate a practical, low-cost improvement toward more truthful LVLM outputs without additional external data or fine-tuning, with broad relevance for reliable multimodal AI systems.

Abstract

Large Visual Language Models (LVLMs) integrate visual and linguistic modalities, exhibiting exceptional performance across various multimodal tasks. Nevertheless, LVLMs remain vulnerable to the issue of object hallucinations. Previous efforts to mitigate this issue focus on supervised fine-tuning (SFT) or incorporating external knowledge, both of which entail significant costs related to training and the acquisition of external data. To address these challenges, we propose a novel model-agnostic approach termed Internal Fact-based Contrastive Decoding (IFCD), designed to mitigate and suppress hallucinations during the inference process of LVLMs by exploiting the LVLMs' own hallucinations. IFCD is grounded in experimental observations that alterations to the LVLMs' internal representations tend to amplify hallucinations caused by language bias. By contrasting disturbed distribution, IFCD calibrates the LVLMs' output and effectively removes the hallucinatory logits from the final predictions. Experimental results validate that IFCD significantly alleviates both object-level and attribute-level hallucinations while achieving an average 9% accuracy improvement on POPE and 8% accuracy improvement on MME object hallucinations subset compared with direct decoding, respectively.

Figures (9)

• Figure 1: Cases of object hallucinations and effect of IFCD on LLaVA 1.5. Given two images, an LLaVA 1.5 outputs responses with attribute and category hallucinations which IFCD fixes.
• Figure 2: An overview of IFCD. IFCD first edits the internal representation of the LVLMs to construct counterfactual logits for comparison by deliberately injecting hallucinations into the model trained by contrastive learning. These counterfactual logits are utilized to reveal potential hallucinatory tendencies of the LVLMs. Furthermore, the internal representation editing model is employed to actively attenuate a portion of the hallucinatory components within the LVLMs, thereby initiating an improvement in the factual accuracy of its outputs. This process effectively corrects the token from an erroneous token "[Fork]" to an accurate "[Dog]".
• Figure 3: An illustration of editing internal representation amplifying language priors. Given an image depicting three black strawberries, LVLMs assign more preference for more conventional strawberry color, such as "red", with increasing editing strength.
• Figure 4: Perception subset of MME results on LLaVA 1.5. Regular denotes the direct sampling method, whereas ICD refers to the Instruction Contrastive Decoding, VCD refers to the Visual Contrastive Decoding baseline and IFCD is a sampling from our proposed contrastive decoding.
• Figure 5: Comparison IFCD and regular decoding on the ratio of hallucination objects ($\text{CHAIR}_i$) with respect to the number of max tokens. IFCD maintains a low ratio of hallucination objects while increasing the number of objects.