Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
Biao Liu, Ning Xu, Junming Yang, Xin Geng
TL;DR
The work tackles multi-objective alignment for large language models by addressing the burden and inefficiency of manually specified, fixed preference weights. It introduces PRO, a lightweight Preference Orchestrator that maps each input prompt $\mathbf{x}$ to a context-specific weight vector $\boldsymbol{w} \in \Delta^{K-1}$, learned from normalized rewards across $K$ objectives via a KL-based objective $\mathcal{L}_{\text{Pro}}$ and softmax-targets $\boldsymbol{w}^*$. The adapter can be integrated with existing methods (e.g., MoRlhf) and supports offline weight conditioning and online, prompt-specific weighting at inference time, enabling adaptive trade-offs without exhaustive manual tuning. Theoretical analysis shows adaptive weighting reduces the alignment gap relative to fixed weights under standard optimization assumptions, and experiments on Reddit Summary, Helpful Assistant, and Ultrafeedback demonstrate that PRO achieves superior multi-objective performance and general capability compared to baselines. Overall, PRO provides a practical, theoretically grounded, and empirically effective solution for prompt-aware multi-objective alignment in LLMs.
Abstract
While Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks, aligning these models with varying human preferences across multiple objectives remains a significant challenge in practical deployments. Existing multi-objective alignment methods rely on manually specified preference weights, which not only burden users with difficult preference specification tasks but also lead to suboptimal training efficiency due to exploration of irrelevant preference combinations. To alleviate these issues, we propose a novel framework named PRO, i.e., PReference Orchestrator, which features a lightweight preference adapter that automatically infers prompt-specific preference weights during both training and deployment phases. Specifically, the adapter automatically learns appropriate preference weights for each prompt by training on normalized reward scores from multiple reward models for preferred responses, which inherently reflect effective preference balances across objectives. Additionally, We provide theoretical analysis proving that our prompt-aware preference mechanism achieves superior performance compared to fixed preference weights in multi-objective alignment scenarios. Extensive experiments across multiple tasks demonstrate the effectiveness of our method over existing multi-objective alignment approaches.