REWARD CONSISTENCY: Improving Multi-Objective Alignment from a Data-Centric Perspective

TL;DR

This work tackles the persistent conflicts that arise when aligning language models to multiple human preferences. It defines Reward Consistency (RC) as the condition that a winning response outperforms a losing one across all objectives, and provides a gradient-based justification for reduced contention in optimization when RC holds. Building on RC, the authors introduce Reward Consistency Sampling (RCS), a data-generation framework that constructs RC-compliant datasets by sampling responses, applying RC filters, and selecting maximal-gap pairs, compatible with both implicit and explicit reward signals. Empirical results show that training with RC-derived data yields substantial gains in harmlessness and helpfulness, and scales to three-objective scenarios including truthfulness, while ablations confirm the necessity of RC and the flexibility of the framework. Overall, RC-RCS offers a practical, data-centric approach to mitigate cross-objective conflicts in direct preference alignment, with notable improvements across representative two- and three-objective setups and clear pathways for future extension.

Abstract

Multi-objective preference alignment in language models often encounters a challenging trade-off: optimizing for one human preference (e.g., helpfulness) frequently compromises others (e.g., harmlessness) due to the inherent conflicts between competing objectives. While prior work mainly focuses on algorithmic solutions, we explore a novel data-driven approach to uncover the types of data that can effectively mitigate these conflicts. Specifically, we propose the concept of Reward Consistency (RC), which identifies samples that align with multiple preference objectives, thereby reducing conflicts during training. Through gradient-based analysis, we demonstrate that RC-compliant samples inherently constrain performance degradation during multi-objective optimization. Building on these insights, we further develop Reward Consistency Sampling, a framework that automatically constructs preference datasets that effectively mitigate conflicts during multi-objective alignment. Our generated data achieves an average improvement of 13.37% in both the harmless rate and helpfulness win rate when optimizing harmlessness and helpfulness, and can consistently resolve conflicts in varying multi-objective scenarios.

Figures (5)

• Figure 1: Gradient analysis of reward consistency.
• Figure 2: Overall pipeline of our proposed RCS framework. While samples in the original preference dataset $D_k$ contain only text for optimizing helpfuless, the samples in our generated dataset $D'_k$ also contain text for optimizing harmlessness, thereby ensuring improvement in both objectives.
• Figure 3: Impact of reward models. RCS performs well using both implicit and explicit reward models.
• Figure 4: Effects of sampling number. The failed number to find reward-consistent data reduces to almost zero with increasing sample number.
• Figure 5: The evaluation prompt for helpfulness.