The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?

TL;DR

This work critiques contrastive decoding methods (VCD, ICD, SID) for mitigating object hallucinations in multimodal LLMs, arguing that reported gains on the POPE Benchmark arise from two misleading factors: a unidirectional bias toward 'Yes' ($\mathcal{R}_1$) and an adaptive plausibility constraint that renders sampling nearly greedy ($\mathcal{R}_2$). By constructing spurious improvement methods (e.g., forced distribution tweaks) and isolating the adaptive constraint, the authors demonstrate that these gains do not reflect true hallucination mitigation. They show that similar improvements can be achieved without addressing hallucinations, challenging common assumptions about the effectiveness of contrastive decoding. The paper calls for more rigorous evaluation that accounts for decoding strategy effects and preserves correct predictions, aiming to steer research toward genuinely robust hallucination mitigation in MLLMs.

Abstract

Contrastive decoding strategies are widely used to reduce object hallucinations in multimodal large language models (MLLMs). These methods work by constructing contrastive samples to induce hallucinations and then suppressing them in the output distribution. However, this paper demonstrates that such approaches fail to effectively mitigate the hallucination problem. The performance improvements observed on POPE Benchmark are largely driven by two misleading factors: (1) crude, unidirectional adjustments to the model's output distribution and (2) the adaptive plausibility constraint, which reduces the sampling strategy to greedy search. To further illustrate these issues, we introduce a series of spurious improvement methods and evaluate their performance against contrastive decoding techniques. Experimental results reveal that the observed performance gains in contrastive decoding are entirely unrelated to its intended goal of mitigating hallucinations. Our findings challenge common assumptions about the effectiveness of contrastive decoding strategies and pave the way for developing genuinely effective solutions to hallucinations in MLLMs.

Figures (7)

• Figure 1: An illustration of hallucination mitigation methods: Visual Contrastive Decoding, Instruction Contrastive Decoding, and Self-Introspective Decoding. The hallucination induction module shifts outputs toward negative responses, while the contrastive decoding module shifts them toward positive responses, rather than achieving their intended effects.
• Figure 2: Changes in the distribution of predictions after applying contrastive decoding methods.
• Figure 3: Skewness in the Raw Output Distribution of MLLMs across Different Datasets
• Figure 4: Why does the adaptive plausibility constraint alone result in improvements?
• Figure 5: Illustration of Prompt-Based Adjustment and Output Layer Modification Algorithms.