CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs

TL;DR

This work targets hallucinations in Multimodal Large Language Models by introducing CHiP, a Cross-modal Hierarchical Direct Preference Optimization framework. CHiP combines Visual Preference Optimization with Hierarchical Textual Preference Optimization to align image and text representations at multiple granularities (response, segment, token) and across modalities. Empirical results on ObjHalBench and other benchmarks show substantial reductions in hallucinations relative to standard DPO and competitive performance versus strong baselines, with ablations confirming the complementary value of HDPO and visual signals. The approach yields improved image-text alignment, more faithful grounding, and practical resources (datasets and code) to advance robust multimodal alignment research.

Abstract

Multimodal Large Language Models (MLLMs) still struggle with hallucinations despite their impressive capabilities. Recent studies have attempted to mitigate this by applying Direct Preference Optimization (DPO) to multimodal scenarios using preference pairs from text-based responses. However, our analysis of representation distributions reveals that multimodal DPO struggles to align image and text representations and to distinguish between hallucinated and non-hallucinated descriptions. To address these challenges, in this work, we propose a Cross-modal Hierarchical Direct Preference Optimization (CHiP) to address these limitations. We introduce a visual preference optimization module within the DPO framework, enabling MLLMs to learn from both textual and visual preferences simultaneously. Furthermore, we propose a hierarchical textual preference optimization module that allows the model to capture preferences at multiple granular levels, including response, segment, and token levels. We evaluate CHiP through both quantitative and qualitative analyses, with results across multiple benchmarks demonstrating its effectiveness in reducing hallucinations. On the Object HalBench dataset, CHiP outperforms DPO in hallucination reduction, achieving improvements of 52.7% and 55.5% relative points based on the base model Muffin and LLaVA models, respectively. We make all our datasets and code publicly available: https://github.com/LVUGAI/CHiP.

Figures (9)

• Figure 1: Comparison of representation distributions and performance across models. Representations are constructed by selecting 150 samples (images, non-hallucinated descriptions, and hallucinated descriptions). The image or text semantics are represented using the last token embedding from the LLM. (d) is the hallucination rate (lower the better) comparison of different models on hallucination benchmarks, namely ObjHal, MMHal, and AMBER. Findings: (1) DPO struggles to align image and description representations and to effectively distinguish between hallucinated and non-hallucinated descriptions. (2) The proposed CHiP method, which incorporates both image and fine-grained text preferences, achieves better alignment between images and ground-truth descriptions while increasing the distance between ground-truth and hallucinated descriptions. (3) CHiP outperforms DPO and original LLaVA in terms of hallucination rate.
• Figure 2: Comparison of preference optimization in different scenarios: (a) DPO, (b) Multimodal DPO, and (c) Cross-modal Hierarchical Direct Preference Optimization (CHiP). $x$ represents the instruction. $y_w$ denotes the response preferred by the human over $y_l$. $m_w$ represents the image that is more likely to generate the preferred response $y_w$ than $m_l$. $\bigstar$ ($\blacktriangle$) and $\bigstar$ ($\blacktriangle$) represent the segments (tokens) involved in the hierarchy reward calculation in the preferred and unpreferred responses.
• Figure 3: Human evaluation results on MMHal-Bench (MMHal).
• Figure 4: Results of Muffin+CHiP and LLaVA+CHiP evaluated on the AMBER dataset with different choices of weight $\lambda$ to control the strength of segment-level preference optimization. Findings: When $\lambda = 1$ ($\lambda = 3$), the best performance of the CHAIR and Hallucination Rate metric is achieved on AMBER based on Muffin (LLaVA-1.6).
• Figure 5: Examples of rejection images constructed by different strategies. (a) is the chosen image.