Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models

TL;DR

The paper tackles multi-objective alignment for generative models by introducing MOPO, an offline algorithm that learns from pairwise human preferences to maximize a primary objective while enforcing tunable lower bounds on secondary objectives through a constrained KL-regularized framework. MOPO derives a dual formulation with an optimal importance-sampling ratio $\rho^*(y)$ and introduces a lower-bound mechanism to robustly estimate constrained objectives, followed by policy extraction via importance-weighted behavioral cloning. Empirical results on synthetic benchmarks show Pareto-front recovery, while real-world experiments with tiny LLMs demonstrate competitive Pareto fronts and resilience to hyperparameters through lagged reference updates and adaptive constraint scheduling. The approach avoids scalarizing all objectives, eliminates the need for pointwise rewards, and delivers scalable, preference-driven multi-objective optimization for large language models. Overall, MOPO enables principled, interactive-friendly alignment across multiple human objectives with provable Pareto-optimality under practical offline data conditions.

Abstract

Post-training of LLMs with RLHF, and subsequently preference optimization algorithms such as DPO, IPO, etc., made a big difference in improving human alignment. However, all such techniques can only work with a single (human) objective. In practice, human users have multiple objectives, such as helpfulness and harmlessness, and there is no natural way to aggregate them into a single objective. In this paper, we address the multi-objective preference-alignment problem, where a policy must optimize several, potentially conflicting, objectives. We introduce the Multi-Objective Preference Optimization (MOPO) algorithm, which frames alignment as a constrained KL-regularized optimization: the primary objective is maximized while secondary objectives are lower-bounded by tunable safety thresholds. Unlike prior work, MOPO operates directly on pairwise preference data, requires no point-wise reward assumption, and avoids heuristic prompt-context engineering. The method recovers policies on the Pareto front whenever the front is attainable; practically, it reduces to simple closed-form iterative updates suitable for large-scale training. On synthetic benchmarks with diverse canonical preference structures, we show that MOPO approximates the Pareto front. When fine-tuning a 1.3B-parameter language model on real-world human-preference datasets, MOPO attains higher rewards and yields policies that Pareto-dominate baselines; ablation studies confirm optimization stability and robustness to hyperparameters.

Figures (11)

• Figure 1: Illustration of how a COP approach, and hence MOPO, achieves Pareto-optimal alignment.
• Figure 2: Illustration of how constraint threshold initialization affects COP solutions.
• Figure 3: MOPO: Impact of preference distribution (gray) over output space with lower-magnitude outputs being preferred with probability 1. $\pi_{\text{lb}}$ constrains lower bound of $\bm{{\mathcal{G}}}(\rho)$, while $\pi_{\text{typ}}$ constrains $\bm{{\mathcal{G}}}(\rho)$ directly (typically).
• Figure 4: Learning Curves of Action Probabilities of MOPO on various dataset types (read column-wise). Shaded region around mean line represents 1 standard deviation over 5 independent runs.
• Figure 5: Results of the Helpful Assistant task and the Reddit Summary task with normalized rewards.

Theorems & Definitions (21)

• Proposition 3 .0
• Proposition 3 .0
• Definition A .1
• Remark A .1
• Definition A .1: $f$-divergence fdivergencefdivergencecsiszar1fdivergencecsiszar2
• Definition A .2: Barrier function convexoptimization
• Definition A .3: Expected calibration error guo2017calibrationfDPO
• Theorem A .4
• Remark A .5: Motivating example
• proof