AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimization

TL;DR

The paper tackles hallucination in multimodal LLMs by improving alignment with visual content. It introduces AdaViP, combining vision-based preference pair construction using RAM, Grounding DINO, SAM, and LAMA with an adaptive optimization that unifies visual and linguistic signals under a Plackett-Luce-like objective. Key contributions include a robust vision-based perturbation pipeline, an adaptive weighting scheme based on implicit rewards, and strong empirical gains on Object HalBench, AMBER, and MMHal-Bench, including AdaViP-7B achieving 93.7% and 96.4% reductions in hallucinations on Object HalBench. The results advance trustworthiness of MLLMs and demonstrate a scalable path toward robust, vision-aware alignment for multimodal tasks.

Abstract

Preference alignment through Direct Preference Optimization (DPO) has demonstrated significant effectiveness in aligning multimodal large language models (MLLMs) with human preferences. However, existing methods focus primarily on language preferences while neglecting the critical visual context. In this paper, we propose an Adaptive Vision-enhanced Preference optimization (AdaViP) that addresses these limitations through two key innovations: (1) vision-based preference pair construction, which integrates multiple visual foundation models to strategically remove key visual elements from the image, enhancing MLLMs' sensitivity to visual details; and (2) adaptive preference optimization that dynamically balances vision- and language-based preferences for more accurate alignment. Extensive evaluations across different benchmarks demonstrate our effectiveness. Notably, our AdaViP-7B achieves 93.7% and 96.4% reductions in response-level and mentioned-level hallucination respectively on the Object HalBench, significantly outperforming current state-of-the-art methods.

Figures (5)

• Figure 1: Comparison of preference construction methods and optimization strategies. (Top) The preferred and rejected images and responses. Language-based preferences maintain the same image ($I_w$) while varying responses ($y_w$ and $y_l$); Vision-based preferences keep the same response ($y_w$) while varying images ($I_w$ and $I_l$). (Bottom) Model log probabilities for different instances under different optimization approaches: base model (No-DPO), language-based DPO, fixed-weight vision-language DPO (No-Adaptive), and our adaptive strategy.
• Figure 2: Overview of our Adaptive Vision-aware Preference optimization (AdaViP). Given a preferred sample, the vision-based rejected one is constructed by locating and removing key elements of the image (Left). Subsequently, to effectively balance the vision- and language-based preference pair, we propose an adaptive loss that modulates the optimization procedure based on the relative rewards (right).
• Figure 3: Visualization results of our vision-based rejected image construction, where we present the rejected image and the candidate set, together with the segmentation set of the image.
• Figure 4: Performance comparisons of the fine-grained metrics in AMBER bench, where $\text{F1}_{\text{A}}$, $\text{F1}_{\text{R}}$, $\text{F1}_{\text{E}}$ represents the Attribute, Relation, and Existence hallucination on the discriminative benchmark, respectively. $\text{F1}$ measures the overall F1 scores on the discriminative benchmark, and $\text{AMBER Score}$ reflects the average performance on both generative and discriminative benchmarks.
• Figure 5: The dynamic weight between the vision- and language-based preferences and the corresponding reward dynamics for different samples during the training procedure.