What Matters in Data for DPO?
Yu Pan, Zhongze Cai, Guanting Chen, Huaiyang Zhong, Chonghuan Wang
TL;DR
This work analyzes what aspects of preference data matter for Direct Preference Optimization (DPO) in aligning LLMs with human preferences. By combining theoretical analysis with controlled experiments, it shows that the quality of chosen responses predominantly drives DPO performance, while the quality of rejected responses has a more limited impact. The authors derive a closed-form optimal policy for DPO under BT assumptions and demonstrate that online DPO with fixed chosen samples effectively conducts supervised fine-tuning on the chosen set, with contrastiveness mainly helping by elevating chosen sample quality. These findings yield practical guidance for constructing high-impact preference datasets and clarify when on-policy data and preference gaps provide meaningful gains.
Abstract
Direct Preference Optimization (DPO) has emerged as a simple and effective approach for aligning large language models (LLMs) with human preferences, bypassing the need for a learned reward model. Despite its growing adoption, a fundamental question remains open: what characteristics of preference data are most critical for DPO performance? In this work, we provide a systematic study of how preference data distribution influences DPO, from both theoretical and empirical perspectives. We show that the quality of chosen responses plays a dominant role in optimizing the DPO objective, while the quality of rejected responses may have relatively limited impact. Our theoretical analysis characterizes the optimal response distribution under DPO and reveals how contrastiveness between responses helps primarily by improving the chosen samples. We further study an online DPO setting and show it effectively reduces to supervised fine-tuning on the chosen responses. Extensive experiments across diverse tasks confirm our findings: improving the quality of chosen responses consistently boosts performance regardless of the quality of the rejected responses. We also investigate the benefit of mixing the on-policy data. Our results interpret the mechanism behind some widely adopted strategies and offer practical insights for constructing high-impact preference datasets for LLM alignment.