New Desiderata for Direct Preference Optimization
Xiangkun Hu, Tong He, David Wipf
TL;DR
The paper addresses the instability and interpolation limitations of direct preference optimization (DPO) and its descendants in aligning large language models with human preferences. It introduces TYPO, a new loss with supervised and unsupervised components designed to preserve strong reference behavior in good regions while improving weak regions, and to satisfy interpolation criteria without relying on RLHF reparameterizations that break under practical constraints. The authors provide theoretical results showing that traditional DPO-family losses cannot simultaneously preserve the BT-optimal policy in favorable prompts and improve elsewhere, and they demonstrate that TYPO achieves BT-like preservation in good regions and SIC interpolation overall. Empirically, TYPO exhibits strong interpolation toward $\pi^*$ in synthetic tests, preserves optimal behavior on favorable prompts, remains robust to optimization constraints, and performs competitively on real-world Anthropic HH data. These findings suggest TYPO as a practical, constraint-robust alternative for aligning LLMs with human preferences, with implications for safer and more reliable alignment in real-world deployments.
Abstract
Large language models in the past have typically relied on some form of reinforcement learning with human feedback (RLHF) to better align model responses with human preferences. However, because of oft-observed instabilities when implementing these RLHF pipelines, various reparameterization techniques have recently been introduced to sidestep the need for separately learning an RL reward model. Instead, directly fine-tuning for human preferences is achieved via the minimization of a single closed-form training objective, a process originally referred to as direct preference optimization (DPO) and followed by several notable descendants. Although effective in certain real-world settings, we introduce new evaluation criteria that serve to highlight unresolved shortcomings in the ability of existing DPO methods to interpolate between a pre-trained reference model and empirical measures of human preferences, as well as unavoidable trade-offs in how low- and high-quality responses are regularized and constraints are handled. Our insights then motivate an alternative DPO-like loss that provably mitigates these limitations. Empirical results serve to corroborate notable aspects of our analyses.