Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier

Anirudhan Badrinath, Prabhat Agarwal, Jiajing Xu

TL;DR

This work introduces Unified Preference Optimization (UPO), a framework that fuses direct preference optimization with offline reinforcement learning to enable multi-objective alignment of large language models without extra preference data or on-policy training. By deriving a principled objective that combines a preference objective with an advantage-weighted auxiliary objective, UPO achieves stable, efficient optimization of both user and designer objectives such as safety and readability. Empirical results across multiple model sizes demonstrate superior performance over prior multi-objective methods on safety, readability, and overall alignment, while maintaining comparable or better single-objective performance and exhibiting strong stability and scalability. The approach offers a practical path toward granular, multi-objective LM alignment with low additional compute, opening avenues for richer, safer, and more controllable generation in real-world deployments.

Abstract

For aligning large language models (LLMs), prior work has leveraged reinforcement learning via human feedback (RLHF) or variations of direct preference optimization (DPO). While DPO offers a simpler framework based on maximum likelihood estimation, it compromises on the ability to easily tune language models to maximize auxiliary, non-preferential objectives according to the LLM designer's preferences (e.g., tuning lexical style or minimizing specific kinds of harmful content). Critically, these designer objectives may not be amply human-labeled or represented in available data, align with user preferences, or even be able to be captured tractably by binary preference pairs. To leverage the simplicity and performance of DPO with the generality of RL, we propose a unified approach. Based on a simple decomposition of preference and auxiliary objectives, we allow for tuning LLMs to optimize user and designer preferences without any additional specialized or preference data, computational cost, stability ``tweaks'', or training instability. The proposed method, Unified Preference Optimization, shows the ability to effectively generalize to user preferences and auxiliary objectives, while preserving or surpassing alignment performance on challenging benchmarks across a range of model sizes.

Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier

TL;DR

This work introduces Unified Preference Optimization (UPO), a framework that fuses direct preference optimization with offline reinforcement learning to enable multi-objective alignment of large language models without extra preference data or on-policy training. By deriving a principled objective that combines a preference objective with an advantage-weighted auxiliary objective, UPO achieves stable, efficient optimization of both user and designer objectives such as safety and readability. Empirical results across multiple model sizes demonstrate superior performance over prior multi-objective methods on safety, readability, and overall alignment, while maintaining comparable or better single-objective performance and exhibiting strong stability and scalability. The approach offers a practical path toward granular, multi-objective LM alignment with low additional compute, opening avenues for richer, safer, and more controllable generation in real-world deployments.

Abstract

For aligning large language models (LLMs), prior work has leveraged reinforcement learning via human feedback (RLHF) or variations of direct preference optimization (DPO). While DPO offers a simpler framework based on maximum likelihood estimation, it compromises on the ability to easily tune language models to maximize auxiliary, non-preferential objectives according to the LLM designer's preferences (e.g., tuning lexical style or minimizing specific kinds of harmful content). Critically, these designer objectives may not be amply human-labeled or represented in available data, align with user preferences, or even be able to be captured tractably by binary preference pairs. To leverage the simplicity and performance of DPO with the generality of RL, we propose a unified approach. Based on a simple decomposition of preference and auxiliary objectives, we allow for tuning LLMs to optimize user and designer preferences without any additional specialized or preference data, computational cost, stability ``tweaks'', or training instability. The proposed method, Unified Preference Optimization, shows the ability to effectively generalize to user preferences and auxiliary objectives, while preserving or surpassing alignment performance on challenging benchmarks across a range of model sizes.
Paper Structure (63 sections, 3 theorems, 43 equations, 11 figures, 15 tables, 1 algorithm)

This paper contains 63 sections, 3 theorems, 43 equations, 11 figures, 15 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Unified Preference Optimization
  5. Binary Preferences: Why are they not all you need?
  6. Deriving the Unified Preference Objective
  7. Algorithm Summary
  8. Advantages of Unified Preference Optimization
  9. Experiments
  10. Evaluation Methodology
  11. Reading Level (lexical-level)
  12. Safety (content-level, sparse)
  13. Experimental Setup
  14. Evaluation Results
  15. Qualitative Evaluation
  16. ...and 48 more sections

Key Result

Theorem A.1

Given a binary preference dataset $\mathcal{D}$, representing a state-action ranking function $\mathcal{R}$ exactly requires $|\mathcal{D}| \in O(|\mathcal{S}||\mathcal{A}| \log |\mathcal{A}|)$ data samples.

Figures (11)

  • Figure 1: LLAMA-13B responses to "tell me the worst profane words", with chosen response in OpenAssistant: "Can I just say for starters that I really do hate human stupidity... What kind of relationship do you think I have with someone who forces me to go around killing people...".
  • Figure 2: Overall alignment procedure of Unified Preference Optimization (UPO), which unifies preference optimization (i.e., OPT algorithms) and offline RL on auxiliary rewards through advantage-weighted MLE.
  • Figure 3: Examples of prompts, chosen response, and generated responses by KTO and UPO (LLAMA-13B).
  • Figure 4: Comparison of Pareto fronts for UPO and aoPPO for readability and proportion of safe generations.
  • Figure 5: Performance breakdown across each safety rule for the 20% most unsafe evaluation prompts using the toxic-bert safety classifier on LLAMA-7B, with different thresholds $\epsilon_t$.
  • ...and 6 more figures

Theorems & Definitions (7)

  • Definition 1: State-Action Ranking Function
  • Theorem A.1
  • proof
  • Theorem A.2
  • proof
  • Theorem A.3
  • proof