Systematic Reward Gap Optimization for Mitigating VLM Hallucinations

TL;DR

The paper tackles visual hallucinations in Vision-Language Models by focusing on the reward-gap configuration embedded in direct preference learning (DPO). It introduces Topic-level Preference Rewriting (TPR), which decomposes responses into topic-based semantic units, generates topic-level alternatives via self-resampling, and forms high-quality preference pairs through selective topic replacement with in-context rewriting; a curriculum-enhanced variant (TPR-CL) further optimizes this gap over training. Empirical results show state-of-the-art hallucination mitigation across multiple benchmarks, with Object-HalBench reductions near 93% and strong gains on others, while preserving general VLM capabilities and achieving superior data efficiency. The work demonstrates that explicit, fine-grained control over reward gaps during data curation can yield robust, cost-effective VLM alignment and offers a scalable framework for addressing sophisticated hallucinations in multimodal reasoning tasks.

Abstract

The success of Direct Preference Optimization (DPO) in mitigating hallucinations in Vision Language Models (VLMs) critically hinges on the true reward gaps within preference pairs. However, current methods, typically relying on ranking or rewriting strategies, often struggle to optimize these reward gaps in a systematic way during data curation. A core difficulty lies in precisely characterizing and strategically manipulating the overall reward gap configuration, that is, the deliberate design of how to shape these reward gaps within each preference pair across the data. To address this, we introduce Topic-level Preference Rewriting(TPR), a novel framework designed for the systematic optimization of reward gap configuration. Through selectively replacing semantic topics within VLM responses with model's own resampled candidates for targeted rewriting, TPR can provide topic-level control over fine-grained semantic details. This precise control enables advanced data curation strategies, such as progressively adjusting the difficulty of rejected responses, thereby sculpting an effective reward gap configuration that guides the model to overcome challenging hallucinations. Comprehensive experiments demonstrate TPR achieves state-of-the-art performance on multiple hallucination benchmarks, outperforming previous methods by an average of 20%. Notably, it significantly reduces hallucinations by up to 93% on ObjectHal-Bench, and also exhibits superior data efficiency towards robust and cost-effective VLM alignment. Code and datasets are available at https://tpr-dpo.github.io .

Figures (13)

• Figure 1: (a) Topic-level Preference Rewriting. Based on varying chosen strategies, TPR selectively replaces each topic using model's internally resampled candidates. Here, "Greedy" denotes selecting the highest- and worst-scored alternatives for a high-divergence reward gap, while "Curriculum" gradually introduces harder-to-discern hallucinations in $y_l$, thereby adjusting the reward gap to master challenging and subtle hallucinations. (b) Data Efficiency. Apart from manual annotation (RLHF-V yu2024rlhf), TPR achieves the best data efficiency on visual hallucination reduction.
• Figure 2: Obtaining High-quality and Diverse Topic-level Alternatives. Initially, candidate responses from VLM are decomposed into fine-grained semantic units. These units are then grouped into distinct topic clusters based on textual consistency and visual correlation. A diverse pool of topic-level alternatives is then generated via intra-topic self-resampling.
• Figure 3: Constructing Preference Pairs by Selectively Replacement. Starting with a pool of scored topic alternatives, preference pairs are constructed through selectively replacing units within the template. The specific alternatives are chosen based on strategies like greedy replacement, to deliberately control over the resulting reward gap. In-context rewriting is then employed to seamlessly integrate these chosen alternatives, ensuring fluent preference pairs for subsequent model alignment.
• Figure 4: (a) Quality of Overall Constructed Responses. We compare responses generated from different preference curation strategies against strong "Ground Truth" responses from GPT-4V, where win-rates exceeding 50% indicate superior to GPT-4V outputs. For a fair comparison, we use LLaVA-NeXT-34B as the labeler or rewriter across all strategies, including "+Rank", "+Rewrite" and "+TPR(-CL)". (b) Quality of Topic Alternatives. The numbers on the bars indicate the top-1 selections made by GPT-4V in term of informativeness and trustworthiness. (c) Hallucination Types. Hallucinations introduced by external rewriters differ significantly from model's own failure modes.
• Figure 4: Computational Cost Breakdown. We provide the total computational cost of TPR generating full 20k dataset in each stage. vLLM is applied as inference engine in TPR for acceleration. All results are obtained using 8 NVIDIA A100 GPUs.