Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucination

Yuanshuai Li, Yuping Yan, Jirui Han, Fei Ming, Lingjuan Lv, Yaochu Jin

TL;DR

IRIS addresses multimodal hallucination by exploiting implicit rewards in the model’s native log-probability space to reveal internal modal conflicts. It eliminates reliance on external evaluators by harvesting self-generated preference pairs through Iterative RVG-guided on-policy sampling and optimization via Direct Preference Optimization. The approach combines a warm-up SFT phase for visual grounding with a grounded, multi-component objective that includes textual and visual preferences plus anchored regularization, achieving strong grounding with only 5.7k samples and demonstrating data efficiency and robustness. Practically, IRIS offers a lightweight, principled path to reduce hallucinations in multimodal LLMs while significantly lowering curation costs compared to external-feedback baselines.

Abstract

Hallucination remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While Direct Preference Optimization (DPO) is a key alignment framework, existing approaches often rely heavily on costly external evaluators for scoring or rewriting, incurring off-policy learnability gaps and discretization loss. Due to the lack of access to internal states, such feedback overlooks the fine-grained conflicts between different modalities that lead to hallucinations during generation. To address this issue, we propose IRIS (Implicit Reward-Guided Internal Sifting), which leverages continuous implicit rewards in the native log-probability space to preserve full information density and capture internal modal competition. This on-policy paradigm eliminates learnability gaps by utilizing self-generated preference pairs. By sifting these pairs based on multimodal implicit rewards, IRIS ensures that optimization is driven by signals that directly resolve modal conflicts. Extensive experiments demonstrate that IRIS achieves highly competitive performance on key hallucination benchmarks using only 5.7k samples, without requiring any external feedback during preference alignment. These results confirm that IRIS provides an efficient and principled paradigm for mitigating MLLM hallucinations.

IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucination

TL;DR

IRIS addresses multimodal hallucination by exploiting implicit rewards in the model’s native log-probability space to reveal internal modal conflicts. It eliminates reliance on external evaluators by harvesting self-generated preference pairs through Iterative RVG-guided on-policy sampling and optimization via Direct Preference Optimization. The approach combines a warm-up SFT phase for visual grounding with a grounded, multi-component objective that includes textual and visual preferences plus anchored regularization, achieving strong grounding with only 5.7k samples and demonstrating data efficiency and robustness. Practically, IRIS offers a lightweight, principled path to reduce hallucinations in multimodal LLMs while significantly lowering curation costs compared to external-feedback baselines.

Abstract

Hallucination remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While Direct Preference Optimization (DPO) is a key alignment framework, existing approaches often rely heavily on costly external evaluators for scoring or rewriting, incurring off-policy learnability gaps and discretization loss. Due to the lack of access to internal states, such feedback overlooks the fine-grained conflicts between different modalities that lead to hallucinations during generation. To address this issue, we propose IRIS (Implicit Reward-Guided Internal Sifting), which leverages continuous implicit rewards in the native log-probability space to preserve full information density and capture internal modal competition. This on-policy paradigm eliminates learnability gaps by utilizing self-generated preference pairs. By sifting these pairs based on multimodal implicit rewards, IRIS ensures that optimization is driven by signals that directly resolve modal conflicts. Extensive experiments demonstrate that IRIS achieves highly competitive performance on key hallucination benchmarks using only 5.7k samples, without requiring any external feedback during preference alignment. These results confirm that IRIS provides an efficient and principled paradigm for mitigating MLLM hallucinations.
Paper Structure (53 sections, 3 theorems, 24 equations, 11 figures, 13 tables, 2 algorithms)

This paper contains 53 sections, 3 theorems, 24 equations, 11 figures, 13 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Hallucination Mitigation in MLLMs
  4. Self-Alignment via DPO Implicit Rewards
  5. Preliminaries
  6. Supervised Fine-Tuning (SFT)
  7. Direct Preference Optimization (DPO).
  8. Methodology
  9. Theoretical Motivation
  10. Learnability of Noisy Self-Generated Preferences.
  11. Implicit Reward Calibration via SFT.
  12. Warm-up and On-policy Self-Generation
  13. Implicit Reward Scoring and Preference Pair Construction
  14. Grounded Preference Learning Objectives
  15. Conditional Textual Preference
  16. ...and 38 more sections

Key Result

Lemma 5.1

Consider the pairwise preference loss where with $\beta>0$ and a fixed reference policy $\pi_{\text{ref}}$. Then the gradient satisfies where $w_\theta(c,y_w,y_l)=\beta\,\sigma\!(-\Delta_\theta(c,y_w,y_l))\in(0,\beta)$.

Figures (11)

  • Figure 1: Overview of IRIS.(a) Comparison between external discrete rewards and IRIS implicit rewards. (b) Empirical hallucination reduction achieved by self-generated preference optimization compared to human-annotated data under the same experimental setting. (c) Conceptual illustration of policy evolution under KL constraints: external correction fails to cross the KL-infeasible region, while IRIS progressively advances the KL-feasible region via self-generated iterations.
  • Figure 2: Comparison of Preference Construction Pipelines.Top: Feedback from proprietary models (e.g., GPT-4). Middle: Prompt-based scoring using large open-source models. Bottom:IRIS (Ours), which leverages intrinsic implicit rewards and Rectified Visual Guidance (RVG) to sift on-policy preference pairs without external evaluators.
  • Figure 3: Effect of Rectification Strength $\gamma$. Sensitivity of hallucination metrics to $\gamma$ across two iterative rounds. The star indicates the optimal value at $\gamma=0.7$.
  • Figure 4: Effect of Visual Preference Weight $\lambda$. Sensitivity of hallucination metrics to the weight $\lambda$ in the final round. The star indicates the optimal value at $\lambda=1.0$.
  • Figure 5: Example of qualitative analysis for Round 3. An example of the model's performance at the 3rd iteration in reducing descriptive illusions and maintaining visual consistency.
  • ...and 6 more figures

Theorems & Definitions (6)

  • Lemma 5.1: Pairwise Gradient Difference Form
  • proof
  • Proposition 5.3: Local Margin Improvement
  • proof
  • Proposition 5.4: Extrema Selection Amplifies the Expected Quality Gap
  • proof : Proof sketch