Primal-Dual Direct Preference Optimization for Constrained LLM Alignment
Yihan Du, Seo Taek Kong, R. Srikant
TL;DR
The paper tackles constrained alignment of LLMs by maximizing a reward signal while keeping harmful content under a threshold. It introduces PD-DPO, which first learns a reward-informed policy via standard DPO and then uses a rearranged Lagrangian DPO on cost data guided by that policy, avoiding explicit reward/cost models or prior knowledge of the optimal multiplier. The authors provide suboptimality and constraint-violation guarantees and extend the approach online with exploration bonuses, removing dependence on offline data coverage. Empirical results on PKU-SafeRLHF show PD-DPO achieves strong helpfulness with improved safety relative to baselines, while offering substantial memory and computation advantages over model-based safety methods. The work thus delivers a theoretically sound, scalable framework for constrained LLM alignment with practical online extensions.
Abstract
The widespread application of Large Language Models (LLMs) imposes increasing demands on safety, such as reducing harmful content and fake information, and avoiding certain forbidden tokens due to rules and laws. While there have been several recent works studying safe alignment of LLMs, these works either require the training of reward and cost models and incur high memory and computational costs, or need prior knowledge about the optimal solution. Motivated by this fact, we study the problem of constrained alignment in LLMs, i.e., maximizing the output reward while restricting the cost due to potentially unsafe content to stay below a threshold. For this problem, we propose a novel primal-dual DPO approach, which first trains a model using standard DPO on reward preference data to provide reward information, and then adopts a rearranged Lagrangian DPO objective utilizing the provided reward information to fine-tune LLMs on cost preference data. Our approach significantly reduces memory and computational costs, and does not require extra prior knowledge. Moreover, we establish rigorous theoretical guarantees on the suboptimality and constraint violation of the output policy. We also extend our approach to an online data setting by incorporating exploration bonuses, which enables our approach to explore uncovered prompt-response space, and then provide theoretical results that get rid of the dependence on preference data coverage. Experimental results on the widely-used preference dataset PKU-SafeRLHF demonstrate the effectiveness of our approach.