Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs

TL;DR

Uni-DPO tackles the inefficiency of Direct Preference Optimization by introducing a unified dynamic preference learning framework that jointly accounts for data quality and the model’s learning dynamics. It adds dual weighting (quality and performance) and a calibrated NLL loss to the DPO objective, along with length normalization, enabling fine-grained credit assignment and stability. Across textual understanding, mathematical reasoning, and multimodal tasks, Uni-DPO delivers consistent gains and scalable improvements, including surpassing strong baselines on Arena-Hard for Gemma-2-9B-IT and significant gains on eight math benchmarks as model size increases. The approach demonstrates robust generalization, improved training stability, and practical reproducibility, highlighting its potential to advance RLHF-based alignment of large language models.

Abstract

Direct Preference Optimization (DPO) has emerged as a cornerstone of reinforcement learning from human feedback (RLHF) due to its simplicity and efficiency. However, existing DPO-based methods typically treat all preference pairs equally, overlooking substantial variations in data quality and learning difficulty, which leads to inefficient data utilization and suboptimal performance. To address this limitation, we propose Uni-DPO, a unified dynamic preference optimization framework that jointly considers (a) the inherent quality of preference pairs and (b) the model's evolving performance during training. By adaptively reweighting samples based on both factors, Uni-DPO enables more effective use of preference data and achieves superior performance. Extensive experiments across models and benchmarks demonstrate the effectiveness and generalization of Uni-DPO. On textual tasks, Gemma-2-9B-IT fine-tuned with Uni-DPO surpasses the leading LLM, Claude 3 Opus, by 6.7 points on Arena-Hard. On mathematical and multimodal tasks, Uni-DPO consistently outperforms baseline methods across all benchmarks, providing strong empirical evidence of its effectiveness and robustness.

Figures (16)

• Figure 1: Demonstrations of data quality and its effect. During preference optimization, high-quality sample pairs exhibit clear distinctions and effectively reflect human preferences, whereas low-quality pairs have ambiguous differences between positive and negative samples, failing to represent human preferences accurately. Therefore, it is crucial to consider data quality when optimizing LLMs with preference data.
• Figure 2: Comparison of DPO and Uni-DPO objectives. The Uni-DPO objective introduces a dual-perspective weighting mechanism, including a quality-aware weight $w_\text{qual}$, a performance-based weight $w_\text{perf}$, and a calibrated negative log-likelihood term $\mathcal{L}_{\text{c-NLL}}$ that emphasizes challenging and high-quality positive samples. Left: schematic illustration of the two objectives. Right: compared with DPO, Uni-DPO dynamically reweights data pairs during training, guiding the optimization trajectory toward regions that better reflect human preference.
• Figure 3: Analysis of reward margin versus score margin. (a) Illustrative reward margin versus score margin examples across training data. The four quadrants reflect the combinations of high/low data quality and easy/hard learning difficulty. (b) Distribution of reward margin (y-axis) versus score margin (x-axis), along with their Spearman correlation coefficient $\rho$, indicating data quality is not necessarily aligned with learning difficulty, suggesting the need to account for both factors during training. (c) Demonstration of dual-perspective weighting mechanism: the quality weight $w_{\text{qual}}$ (top) rises with data quality, while the performance weight $w_{\text{perf}}$ (bottom) decreases as model performance improves, ensuring updates target both high-quality yet under-fitted samples.
• Figure 4: Illustration of model performance weighting. (a) Focal-weighted DPO gradients are larger for hard examples and decrease as the model improves, encouraging the focus on harder examples. (b) Direct focal weighting (\ref{['eq:w_focal']}) can lead to unstable training, whereas our stabilized form (\ref{['eq:w_performance']}) applies a uniform constraint on each sample, resulting in more stable training. (c)-(d) Incorporating $w_{\text{perf}}$ can reduce overfitting.
• Figure 5: Parameter sensitivity and scaling consistency. (a)–(c) Sensitivity of the performance weight $w_{\text{perf}}$ (\ref{['eq:w_performance']}) hyperparameters $\gamma$ and $\tau_{\text{ref}}$: (a),(b) show practical ranges of $[1.0,5.0]$ for $\gamma$ and $[0.5,2.0]$ for $\tau_{\text{ref}}$; (c) plots the optimal $\tau_{\text{ref}}$ and corresponding Arena-Hard WR scores across different $\gamma$ values, revealing that higher $\gamma$ indicates sharper gradient decrease on easy sample, requires stronger margin constraints $\tau_{\text{ref}}$. (d),(e) Uni-DPO consistently outperforms SimPO on Qwen2.5 base model across different sizes, revealing its scalability.