Benchmarking Direct Preference Optimization for Medical Large Vision-Language Models
Dain Kim, Jiwoo Lee, Jaehoon Yun, Yong Hoe Koo, Qingyu Chen, Hyunjae Kim, Jaewoo Kang
TL;DR
The paper critically evaluates Direct Preference Optimization (DPO) for medical large vision-language models by benchmarking nine DPO variants on two medical LVLMs (LLaVA-Med and HuatuoGPT-Vision) across VQA, image captioning, and radiology report generation. It finds that gains from DPO are often modest and comparable to supervised fine-tuning, and that a pervasive problem is visual misinterpretation that many DPO methods only partially mitigate. To address this, the authors propose a targeted, error-type aware DPO approach that explicitly counters four common visual errors, achieving a notable 3.6% improvement over the strongest existing DPO baseline on VQA. They further validate through expert evaluation that visual grounding remains a core bottleneck and demonstrate robust gains with domain-targeted DPO, while releasing their full training framework, data, and code to accelerate future research in medically reliable LVLMs.
Abstract
Large Vision-Language Models (LVLMs) hold significant promise for medical applications, yet their deployment is often constrained by insufficient alignment and reliability. While Direct Preference Optimization (DPO) has emerged as a potent framework for refining model responses, its efficacy in high-stakes medical contexts remains underexplored, lacking the rigorous empirical groundwork necessary to guide future methodological advances. To bridge this gap, we present the first comprehensive examination of diverse DPO variants within the medical domain, evaluating nine distinct formulations across two medical LVLMs: LLaVA-Med and HuatuoGPT-Vision. Our results reveal several critical limitations: current DPO approaches often yield inconsistent gains over supervised fine-tuning, with their efficacy varying significantly across different tasks and backbones. Furthermore, they frequently fail to resolve fundamental visual misinterpretation errors. Building on these insights, we present a targeted preference construction strategy as a proof-of-concept that explicitly addresses visual misinterpretation errors frequently observed in existing DPO models. This design yields a 3.6% improvement over the strongest existing DPO baseline on visual question-answering tasks. To support future research, we release our complete framework, including all training data, model checkpoints, and our codebase at https://github.com/dmis-lab/med-vlm-dpo.