ADPO: Anchored Direct Preference Optimization
Wang Zixian
TL;DR
ADPO introduces Anchored Direct Preference Optimization, which minimizes $\mathrm{KL}(q\|\tilde{p}_\theta)$ where $\tilde{p}_\theta=\mathrm{softmax}\big(\frac{\boldsymbol{\ell}-\boldsymbol{\ell}^{\text{ref}}}{\tau_{\text{anc}}}\big)$. By centering logits around a reference policy and using forward KL, ADPO unifies supervised fine-tuning, knowledge distillation, maximum-entropy RL, and DPO as special cases, and reveals an implicit trust region governed by the softmax Fisher metric with curvature $\propto 1/\tau_{\text{anc}}^2$. Empirically, ADPO achieves 12–93% improvements over standard DPO across 12 noisy scenarios, with listwise variants often topping results; offline distillation sees up to 4–49× reductions in student–teacher KL and substantial returns. The framework also identifies task-dependent anchor strategies: dynamic anchors boost online exploration in noisy environments, while fixed anchors stabilize offline distillation. Overall, anchoring emerges as a general principle for robust policy optimization across diverse learning contexts, guiding anchor selection for online RLHF and offline distillation.
Abstract
Direct Preference Optimization (DPO) has emerged as a simple alternative to reinforcement learning from human feedback (RLHF) for aligning language models, but its reliance on hard pairwise labels makes it brittle under noise; our experiments show performance degrading by up to 93 percent in noisy settings. We introduce Anchored Direct Preference Optimization (ADPO), a unified framework that addresses this fragility through reference anchoring. By minimizing KL(q || softmax((l - l_ref) / tau_anc)), where l_ref are reference policy log probabilities, ADPO provides three key advantages: (1) it unifies major learning paradigms, including supervised fine-tuning, knowledge distillation, maximum-entropy reinforcement learning, and DPO, as special cases through different choices of target distribution q, anchor policy pi_ref, and temperature tau_anc; (2) it induces an implicit trust region governed by the softmax Fisher metric with curvature scaling as 1 / tau_anc^2, providing geometric regularization absent in standard methods; and (3) it enables flexible anchor strategies tailored to different learning contexts. Empirically, ADPO consistently outperforms standard DPO by 12 to 93 percent across twelve noisy scenarios, with listwise variants achieving top performance in eleven of twelve cases. In offline distillation, ADPO reduces student-teacher KL by 4 to 49 times while achieving superior returns (for example, 279.3 vs -309.0 for knowledge distillation on HalfCheetah). We further uncover a task-dependent tradeoff: dynamic anchors excel at online exploration in noisy environments (plus 5 to 11 percent), while fixed anchors enable stable offline distillation. Our work establishes anchoring as a general principle for robust policy optimization, with clear practical guidance for anchor selection across diverse learning scenarios.