CRoPS: A Training-Free Hallucination Mitigation Framework for Vision-Language Models

TL;DR

CRoPS introduces a training-free hallucination mitigation framework for vision-language models by generalizing contrastive decoding to combine two specialized hallucinated models: one that suppresses visual tokens to address vision-driven errors and another that trims key textual tokens to counter text-driven biases. The method uses a time-adaptive weighting scheme to emphasize different hallucination sources across generation, achieving robust reductions in hallucination across six benchmarks and three LVLM families with minimal retraining. Empirical results show CRoPS outperforms state-of-the-art training-free methods on CHAIR, AMBER, GPT-4o-assisted, and POPE benchmarks, while maintaining fluency and detail in captions. The work demonstrates that training-free, multi-model contrastive decoding can substantially improve visual grounding in LVLMs, with practical latency overheads comparable to existing approaches.

Abstract

Despite the rapid success of Large Vision-Language Models (LVLMs), a persistent challenge is their tendency to generate hallucinated content, undermining reliability in real-world use. Existing training-free methods address hallucinations but face two limitations: (i) they rely on narrow assumptions about hallucination sources, and (ii) their effectiveness declines toward the end of generation, where hallucinations are most likely to occur. A common strategy is to build hallucinated models by completely or partially removing visual tokens and contrasting them with the original model. Yet, this alone proves insufficient, since visual information still propagates into generated text. Building on this insight, we propose a novel hallucinated model that captures hallucination effects by selectively removing key text tokens. We further introduce Generalized Contrastive Decoding, which integrates multiple hallucinated models to represent diverse hallucination sources. Together, these ideas form CRoPS, a training-free hallucination mitigation framework that improves CHAIR scores by 20% and achieves consistent gains across six benchmarks and three LVLM families, outperforming state-of-the-art training-free methods.

Figures (10)

• Figure 1: Overview of CRoPS framework. CRoPS combines two hallucinated models: one that removes visual tokens to capture vision-related hallucinations, and another that removes key textual tokens to capture text-related hallucinations. Their outputs are then integrated through generalized contrastive decoding framework to reduce hallucinations in LVLMs.
• Figure 2: Left: Plot of dependency measure (see section \ref{['subsec:problem1']}), which quantifies the influence of vision and vision+text tokens on LVLM generation.We observe that $\text{VD}(t)$ decreases over time, indicating that the model relies less on vision tokens as decoding progresses. Right: Frequency of hallucinated objects that frequently co-occur with the ground truth object "dining table". We observe that SID and CRoPS effectively mitigate statistical biases, whereas M3ID performs sub-optimally.
• Figure 3: Comparison of image descriptions from different methods. Vanilla, SID, and M3ID contain hallucinated details (highlighted in red), e.g., animals and exaggerated snow coverage. In contrast, CRoPS produces a faithful description without these hallucinations. Note that hallucinations become more frequent during later stages of generation.
• Figure 4: Left: Plot of Jensen-Shannon (JS) divergence over generation time between different hallucinated models. The dashed blue line indicates $\log(2)$, which is the maximum possible divergence value. Right: Plot of Visual Dependency of final outputs across different methods (Sampling, SID, M3ID, and CRoPS).
• Figure 5: Evaluation on the GPT-4o assisted benchmark zhou2024analyzingmitigatingobjecthallucination, comparing hallucination (SHR), fluency (1- and 2-gram precision), and descriptive detail (WPI and SPI). Larger enclosed areas correspond to better overall performance. Please zoom in for clearer visualization.