DA-DPO: Cost-efficient Difficulty-aware Preference Optimization for Reducing MLLM Hallucinations

TL;DR

This work addresses hallucinations in multimodal large language models by diagnosing overfitting to easy preference samples in Direct Preference Optimization. It introduces DA-DPO, a cost-efficient framework that simultaneously estimates data difficulty with pretrained contrastive and generative vision–language models and applies difficulty-aware reweighting during training. Across hallucination and comprehensive benchmarks, DA-DPO delivers robust reductions in factual errors while preserving or improving general multimodal capabilities, without requiring new data or additional fine-tuning stages. The approach offers practical gains for faithful multimodal reasoning and demonstrates the value of training-free difficulty estimation and adaptive data weighting in preference alignment.

Abstract

Direct Preference Optimization (DPO) has shown strong potential for mitigating hallucinations in Multimodal Large Language Models (MLLMs). However, existing multimodal DPO approaches often suffer from overfitting due to the difficulty imbalance in preference data. Our analysis shows that MLLMs tend to overemphasize easily distinguishable preference pairs, which hinders fine-grained hallucination suppression and degrades overall performance. To address this issue, we propose Difficulty-Aware Direct Preference Optimization (DA-DPO), a cost-effective framework designed to balance the learning process. DA-DPO consists of two main components: (1) Difficulty Estimation leverages pre-trained vision--language models with complementary generative and contrastive objectives, whose outputs are integrated via a distribution-aware voting strategy to produce robust difficulty scores without additional training; and (2) Difficulty-Aware Training reweights preference pairs based on their estimated difficulty, down-weighting easy samples while emphasizing harder ones to alleviate overfitting. This framework enables more effective preference optimization by prioritizing challenging examples, without requiring new data or extra fine-tuning stages. Extensive experiments demonstrate that DA-DPO consistently improves multimodal preference optimization, yielding stronger robustness to hallucinations and better generalization across standard benchmarks, while remaining computationally efficient. The project page is available at https://artanic30.github.io/project_pages/DA-DPO/.

Figures (7)

• Figure 1: (\ref{['fig:avf_performance']}) Performance Comparison of DPO and DA-DPO. We provide the performance improvements of DPO and DA-DPO compared to the LLaVA v1.5 7B without preference optimization. The Hallucination indicates the performance on 4 hallucination benchmarks, and Comprehensive indicates the performance of 4 comprehensive MLLM benchmarks. The details are described in the experiments section. (\ref{['fig:easy_hard']}) Easy and hard pairwise samples: "Easy Samples" have a large score gap due to clear differences between preferred and dispreferred responses, while "Hard Samples" show minor differences, making them more valuable for learning.
• Figure 2: Reward Dynamics and Area-Under-Gap (AUG) Between the Easiest and Hardest Samples. We present the reward trajectories on a held-out validation set of LLaVA-v1.5-7B for both DPO and DA-DPO trained on the BPO dataset. The first row depicts how the rewards of data buckets with different estimated difficulty levels evolve over training iterations. The difficulty is estimated using three distinct proxies, as detailed in Section \ref{['sec:beta']}. Here, the Easiest samples (Top 75–100% in the legend) correspond to samples with the largest gap between chosen and rejected responses. The second row reports the Area Under Gap (AUG), quantifying the cumulative reward gap between the easiest and hardest samples across training, serving as a compact indicator of reward disparity dynamics. Shaded regions in the plots indicate the standard deviation across three independent seeds, though the effect is subtle due to the low variance induced by training randomness.
• Figure 3: Illustration of the two contrastive and generative VLMs. As shown on the right side, CLIP captures the misalignment between the image and the answer such that the CLIP score gap is large when chosen and rejected answers differ in image relevance. However, as shown on the left side, the MLLM is better at capturing the logical connection between the question and answers, such that the MLLM score gap is large when chosen and rejected answers differ in question relevance.
• Figure 4: Preference classification comparison. Preference classification evaluates whether the pretrained VLMs output a higher reward score for the chosen answer compared with the rejected answer. We report the classification accuracy on three subcategories and the overall performance on the BPO training dataset.
• Figure 5: BPO