Robust Multi-Objective Preference Alignment with Online DPO

TL;DR

MO-ODPO introduces a single, prompt-conditioned policy capable of representing multiple objective weightings along the Pareto frontier for multi-objective LLM alignment. It integrates online Direct Preference Optimization with a prompt-conditioning mechanism and Dirichlet-weight sampling to cover diverse weight combinations without retraining for each point. Through experiments on Anthropic-HH and TL;DR benchmarks, MO-ODPO Pareto-dominates strong baselines and demonstrates robust inference-time steerability, even on smaller PaLM 2 models. The work also analyzes how objective-weight sampling and training dynamics influence frontier quality and mode behavior, offering insights for practical deployment and future enhancements. Overall, this method provides an efficient, scalable path to configurable, safe, and useful AI systems that balance conflicting human preferences.

Abstract

Multi-objective preference alignment of large language models (LLMs) is critical for developing AI systems that are more configurable, personalizable, helpful, and safe. However, optimizing model outputs to satisfy diverse objectives with variable weights at inference time for truly personalized models presents a significant challenge. Existing approaches are either computationally expensive to train or do not sufficiently steer model behaviors. This paper introduces the Multi-Objective Online DPO (MO-ODPO) algorithm, designed to robustly and efficiently align model behaviors with multiple, potentially conflicting human preferences. Our approach incorporates a prompt conditioning mechanism, allowing us to train a single preference-conditional policy, that can adapt to new preference combinations at inference. Experiments on two popular benchmarks show that MO-ODPO Pareto-dominates existing baselines while providing excellent inference-time steerability between diverse objectives.

Figures (9)

• Figure 1: Representation of the MO-ODPO algorithm with two rewards
• Figure 2: Pareto fronts for Anthropic-HH (left) and TL;DR Summarization (right) with PaLM 2 XS policy model
• Figure 3: Pareto fronts for Anthropic-HH (left) and TL;DR Summarization (right) with PaLM 2 XXS policy model
• Figure 4: Automated evaluation win rates for MO-ODPO compared against individual baselines, as judged by a large off-the-shelf LLM. MO-ODPO consistently achieves a higher win rate than each baseline on both benchmarks.
• Figure 5: Pareto fronts for MO-ODPO (PaLM 2 XS) with different $Dirichlet(\alpha)$ sampling of objective weights