mDPO: Conditional Preference Optimization for Multimodal Large Language Models

TL;DR

<3-5 sentence high-level summary> The paper identifies an unconditional visual bias as a key pitfall in applying direct preference optimization (DPO) to multimodal LLMs and proposes mDPO, which adds conditional preference optimization on images and a reward anchor to stabilize chosen-response likelihood. By combining standard DPO with the new CoPO and AnchPO objectives, mDPO forces the model to leverage both visual and linguistic cues and prevents likelihood degradation of preferred responses. Empirical results across two models (Bunny-v1.0-3B and LLaVA-v1.5-7B) and three benchmarks (MMHalBench, Object HalBench, AMBER) show consistent improvements, particularly in reducing hallucinations, and human evaluations corroborate higher quality outputs. The work demonstrates that properly calibrated multimodal preference objectives can outperform purely data-driven scaling, enabling more reliable multimodal alignment across model sizes.

Abstract

Direct preference optimization (DPO) has shown to be an effective method for large language model (LLM) alignment. Recent works have attempted to apply DPO to multimodal scenarios but have found it challenging to achieve consistent improvement. Through a comparative experiment, we identify the unconditional preference problem in multimodal preference optimization, where the model overlooks the image condition. To address this problem, we propose mDPO, a multimodal DPO objective that prevents the over-prioritization of language-only preferences by also optimizing image preference. Moreover, we introduce a reward anchor that forces the reward to be positive for chosen responses, thereby avoiding the decrease in their likelihood -- an intrinsic problem of relative preference optimization. Experiments on two multimodal LLMs of different sizes and three widely used benchmarks demonstrate that mDPO effectively addresses the unconditional preference problem in multimodal preference optimization and significantly improves model performance, particularly in reducing hallucination.

Figures (5)

• Figure 1: We train Bunny-v1.0-3B he2024bunny on 10K multimodal preference data from Silkie li2023silkie with different variants of DPO. We perform DPO (No Image) where all images are removed from the preference data. Counterintuitively, the overall score on the MMHalBench sun2023aligning for DPO (No Image) is similar to that of DPO with images. This finding suggests that DPO may suffer from unconditional preferences, neglecting the visual modality during optimization. Our proposed method, mDPO, effectively addresses this issue and improves model performance.
• Figure 2: Overview of mDPO. Top Left: Standard DPO expects the multimodal LLM to learn response preferences conditioned on both the image and the question. Top Right: However, in practice, the learning process often disregards the image condition. Bottom: To address this issue, mDPO introduces an additional image preference learning objective to emphasize the relationship between the image and the response. Furthermore, mDPO incorporates a reward anchor to ensure that the probability of the chosen response does not decrease.
• Figure 3: Qualitative Results from MMHalBench. Top: When trained with standard DPO, Bunny often assumes the image description in the question is correct, responding accordingly, even if the question contains an adversarial premise regarding the image. In contrast, mDPO identifies the false premise in the question by referencing the image. Bottom: Bunny trained with standard DPO may disregard the image and provide an educated guess for the answer. Conversely, mDPO delivers a correct answer that is conditioned on the image.
• Figure 4: Human evaluation on MMHalBench.
• Figure 5: Impact of data scale on the performance of standard DPO and mDPO, using Bunny as the base model. We assess the overall score and hallucination rate on MMHallBench. mDPO is effective across different scales, whereas standard DPO does not exhibit a scaling effect in multimodal scenarios.