Ground What You See: Hallucination-Resistant MLLMs via Caption Feedback, Diversity-Aware Sampling, and Conflict Regularization

TL;DR

This paper tackles hallucination in multimodal LLMs trained with reinforcement learning by diagnosing three root causes: visual misinterpretation, insufficient exploration diversity, and sample interference via the Neural Tangent Kernel. It introduces a three-part framework: Visual-Grounded Reasoning Enhancement with a Caption Reward to ensure grounding; Reward Variance-Guided Sample Selection to promote informative exploration through high-variance, medium-difficulty samples; and Conflict-Aware Regularization using an InfoNCE loss to regulate NTK-based sample interactions. Empirical results on diverse image/video benchmarks show significant reductions in hallucination and improved inference accuracy, achieving state-of-the-art open-source performance on several tasks and demonstrating the effectiveness of the proposed components. The work provides practical guidance for stabilizing RL optimization in MLLMs by connecting visual grounding, data selection, and gradient-interaction dynamics, with implications for safer and more reliable multimodal reasoning.

Abstract

While Multimodal Large Language Models (MLLMs) have achieved remarkable success across diverse tasks, their practical deployment is severely hindered by hallucination issues, which become particularly acute during Reinforcement Learning (RL) optimization. This paper systematically analyzes the root causes of hallucinations in MLLMs under RL training, identifying three critical factors: (1) an over-reliance on chained visual reasoning, where inaccurate initial descriptions or redundant information anchor subsequent inferences to incorrect premises; (2) insufficient exploration diversity during policy optimization, leading the model to generate overly confident but erroneous outputs; and (3) destructive conflicts between training samples, where Neural Tangent Kernel (NTK) similarity causes false associations and unstable parameter updates. To address these challenges, we propose a comprehensive framework comprising three core modules. First, we enhance visual localization by introducing dedicated planning and captioning stages before the reasoning phase, employing a quality-based caption reward to ensure accurate initial anchoring. Second, to improve exploration, we categorize samples based on the mean and variance of their reward distributions, prioritizing samples with high variance to focus the model on diverse and informative data. Finally, to mitigate sample interference, we regulate NTK similarity by grouping sample pairs and applying an InfoNCE loss to push overly similar pairs apart and pull dissimilar ones closer, thereby guiding gradient interactions toward a balanced range. Experimental results demonstrate that our proposed method significantly reduces hallucination rates and effectively enhances the inference accuracy of MLLMs.

Figures (5)

• Figure 1: Three Types of Hallucination in RL-Tuned Multimodal LLMs: Visual Misinterpretation, Limited Exploration, and Sample Conflict
• Figure 2: The proposed framework for robust visual reasoning, composed of three components: Reward Variance-Guided Sample Selection, Visual-Grounded Reasoning Enhancement, and Conflict-Aware Regularization.
• Figure 3: Caption and Answer Reward curves during RL training on medium and hard samples.
• Figure 4: Policy entropy distribution across models trained on easy, medium, and hard samples.
• Figure 5: (a) Accuracy under different NTK thresholds $\tau$. (b) and (c) Reshaping Effect on Final Layer Sample Representations Before and After InfoNCE Application.