Multi-Value Alignment for LLMs via Value Decorrelation and Extrapolation

TL;DR

Multi-value alignment for LLMs faces interference when optimizing multiple human values. The authors propose MVA, incorporating Value Decorrelation Training that minimizes mutual information between value-specific updates via HSIC, and Value Combination Extrapolating to explore broader Pareto-frontier solutions by composing decorrelated value vectors $\{\theta_i\}$. They demonstrate that decorrelation reduces parameter interference and that extrapolation yields diverse, Pareto-optimal models, outperforming baselines on two benchmark datasets. The approach offers a scalable, plug-and-play path to balance complex safety and alignment objectives in practical LLM deployments.

Abstract

With the rapid advancement of large language models (LLMs), aligning them with human values for safety and ethics has become a critical challenge. This problem is especially challenging when multiple, potentially conflicting human values must be considered and balanced. Although several variants of existing alignment methods (such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO)) have been proposed to address multi-value alignment, they suffer from notable limitations: 1) they are often unstable and inefficient in multi-value optimization; and 2) they fail to effectively handle value conflicts. As a result, these approaches typically struggle to achieve optimal trade-offs when aligning multiple values. To address this challenge, we propose a novel framework called Multi-Value Alignment (MVA). It mitigates alignment degradation caused by parameter interference among diverse human values by minimizing their mutual information. Furthermore, we propose a value extrapolation strategy to efficiently explore the Pareto frontier, thereby constructing a set of LLMs with diverse value preferences. Extensive experiments demonstrate that MVA consistently outperforms existing baselines in aligning LLMs with multiple human values.

Figures (11)

• Figure 1: Overview of MVA framework.
• Figure 2: Pareto Frontiers of MVA and Baselines on Anthropic-HH and BeaverTails. A curve closer to the top right indicates better alignment performance.
• Figure 3: Winrates of MVA against baselines on Anthropic-HH (a,b) and BeaverTails (c,d).
• Figure 4: Comparison of MVA with HSIC and orthogonality constraints. The HSIC-constrained curve lies closer to the top right, demonstrating superior alignment performance.
• Figure 5: Performance of single value vectors.