3D-Properties: Identifying Challenges in DPO and Charting a Path Forward
Yuzi Yan, Yibo Miao, Jialian Li, Yipin Zhang, Jian Xie, Zhijie Deng, Dong Yan
TL;DR
This work investigates why reward-model-free Direct Preference Optimization struggles to reach RM-based RLHF PPO performance. It identifies three instability phenomena, the 3D-properties, arising from the interaction of chosen and rejected gradients, and validates them with a toy model and real LLM tasks in math and instruction-following. It then proposes regularization approaches—Flex-DPO with adaptive beta and SFT-DPO—that mitigate these instabilities and improve stability, especially under on-policy data. By contrasting DPO with RM-based alignment, the authors explain the gap in performance and provide concrete guidance for advancing reward-model-free preference learning toward the efficacy of PPO-based methods. The results offer practical insights into optimization dynamics, data distribution effects, and avenues for future research in alignment without explicit reward models.
Abstract
Aligning large language models (LLMs) with human preferences has gained significant attention, with Proximal Policy Optimization (PPO) as a standard yet computationally expensive method and Direct Preference Optimization (DPO) as a more efficient alternative. While DPO offers simplicity, it remains underutilized in state-of-the-art LLMs, suggesting potential limitations. In this work, we revisit DPO, analyzing its theoretical foundations and empirical performance to bridge this gap. We identify three key properties, termed 3D properties, that emerge from DPO's learning process: Drastic drop in rejected response likelihood, Degradation into response suppression, and Dispersion effect on unseen responses. We show that these issues arise from DPO's optimization dynamics, where the interaction between chosen and rejected response gradients leads to instability. Our findings are supported by experiments on both a controlled toy model and real-world LLM tasks, including mathematical problem-solving and instruction following. To address these challenges, we propose simple regularization techniques that improve training stability and performance. Additionally, we examine how preference data distribution impacts DPO's effectiveness, offering insights into how alignment models handle out-of-domain (OOD) data. Our work connects these observations to broader research and provides a theoretical explanation for DPO's limitations. We hope these insights will guide future advancements in reward-model-free preference learning, bringing it closer to reward-model-based approaches.