SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment

TL;DR

This work tackles the instability and suboptimality of Direct Preference Optimization (DPO) in aligning large language models with human values. It introduces Self-Guided Direct Preference Optimization (SGDPO), which adds a pilot term to steer gradient flow and uses subsequences to refine updates to chosen and rejected rewards. The authors provide theoretical analyses of gradient behavior and demonstrate, across multiple models and benchmarks, that SGDPO delivers more robust and higher-quality alignment than DPO, with improvements up to 9.19% in reported scores. The approach achieves stronger, more consistent performance while incurring only modest computational overhead, indicating practical benefits for scalable alignment of LLMs.

Abstract

Direct Preference Optimization (DPO) is broadly utilized for aligning Large Language Models (LLMs) with human values because of its flexibility. Despite its effectiveness, it has been observed that the capability of DPO to generate human-preferred response is limited and the results of DPO are far from resilient. To address these limitations, in this paper we propose a novel Self-Guided Direct Preference Optimization algorithm, i.e., SGDPO, which incorporates a pilot term to steer the gradient flow during the optimization process, allowing for fine-grained control over the updates of chosen and rejected rewards. We provide a detailed theoretical analysis of our proposed method and elucidate its operational mechanism. Furthermore, we conduct comprehensive experiments on various models and benchmarks. The extensive experimental results demonstrate the consistency between the empirical results and our theoretical analysis and confirm the effectiveness of our proposed approach (up to 9.19% higher score).

Figures (10)

• Figure 1: Reward curves on various base models: (a) DPO reward curves on Llama-3.1 instruct 8B; (b) DPO reward curves on Llama-3.1 base 8B; (c) DPO reward curves on Qwen-2 instruct 7B; (d) DPO reward curves on Qwen-2 base 7B. (e) Our SGDPO reward curves on Llama-3.1 instruct 8B; (f) Our SGDPO reward curves on Llama-3.1 base 8B; (g) Our SGDPO reward curves on Qwen-2 instruct 7B; (h) Our SGDPO reward curves on Qwen-2 base 7B.
• Figure 2: Gradient flow of DPO ($\beta=0.1$) with large values truncated at 1.5.
• Figure 3: Training reward curves for the Llama-3.1 base 8B model using the SGDPO method: (a) $r_1 = 0.6$ and $r_2 = 0.6$. (b) $r_1 = 0.7$ and $r_2 = 0.7$. (c) $r_1 = 0.8$ and $r_2 = 0.8$. (d) $r_1 = 0.9$ and $r_2 = 0.9$. "CCR" represents the average (last 80 iterations) convergence chosen reward and "CRR" represents the average (last 80 iterations) convergence reject reward.
• Figure 4: MT-Bench Results across different model configurations, using Llama-3.1 instruct 8B as the base model.
• Figure 5: MT-Bench results of SGDPO across various configurations, using Llama-3.1 base 8B as the base model. The dashed lines represent the score and the token length of DPO.

Theorems & Definitions (12)

• Theorem 1
• proof
• Theorem 2
• proof
• Theorem 3
• proof
• Theorem 1
• proof
• Theorem 2
• proof