Multi-Objective Alignment of Large Language Models Through Hypervolume Maximization
Subhojyoti Mukherjee, Anusha Lalitha, Sailik Sengupta, Aniket Deshmukh, Branislav Kveton
TL;DR
This work tackles multi-objective alignment of large language models (MOAHF) when human preferences are unknown or hard to quantify. It proposes HaM, an a-posteriori MO approach that learns $K$ diverse LLM policies by directly maximizing the hypervolume across $J$ objectives, ensuring coverage of the Pareto front. Key innovations include a shared-transformer backbone with per-policy heads, mini-batch based hypervolume estimation, and a random hypervolume scalarization option to reduce computational burden. Empirical results across harmlessness, helpfulness, humor, faithfulness, and hallucination demonstrate that HaM yields stronger and more diverse Pareto fronts than baselines like RiC and SCA, with robust performance in both offline and online fine-tuning regimes. Overall, HaM offers a scalable, principled pathway to balance conflicting human preferences in MOAHF while maintaining practical resource requirements.
Abstract
Multi-objective alignment from human feedback (MOAHF) in large language models (LLMs) is a challenging problem as human preferences are complex, multifaceted, and often conflicting. Recent works on MOAHF considered a-priori multi-objective optimization (MOO), where human preferences are known at training or inference time. In contrast, when human preferences are unknown or difficult to quantify, a natural approach is to cover the Pareto front by multiple diverse solutions. We propose an algorithm HaM for learning diverse LLM policies that maximizes their hypervolume. This is the first application of a-posteriori MOO to MOAHF. HaM is computationally and space efficient, and empirically superior across objectives such as harmlessness, helpfulness, humor, faithfulness, and hallucination, on various datasets.