Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment

TL;DR

This work tackles reward misgeneralization caused by noisy human preferences in RM for LLM alignment. It introduces Collaborative Reward Modeling (CRM), an online, data-centric framework where two RMs peer-review each other’s data selections and learn via curriculum learning to filter noisy preferences and synchronize progress. Empirical results across multiple reward-model benchmarks show robust improvements, including up to a 9.94-point gain on RewardBench under 40% noise and consistent gains in both explicit and implicit reward-alignment settings. CRM thus offers a scalable, versatile strategy to improve RM generalization and safer, more effective LLM alignment with human values.

Abstract

Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human values. However, noisy preferences in human feedback can lead to reward misgeneralization - a phenomenon where reward models learn spurious correlations or overfit to noisy preferences, which poses important challenges to the generalization of RMs. This paper systematically analyzes the characteristics of preference pairs and aims to identify how noisy preferences differ from human-aligned preferences in reward modeling. Our analysis reveals that noisy preferences are difficult for RMs to fit, as they cause sharp training fluctuations and irregular gradient updates. These distinctive dynamics suggest the feasibility of identifying and excluding such noisy preferences. Empirical studies demonstrate that policy LLM optimized with a reward model trained on the full preference dataset, which includes substantial noise, performs worse than the one trained on a subset of exclusively high quality preferences. To address this challenge, we propose an online Collaborative Reward Modeling (CRM) framework to achieve robust preference learning through peer review and curriculum learning. In particular, CRM maintains two RMs that collaboratively filter potential noisy preferences by peer-reviewing each other's data selections. Curriculum learning synchronizes the capabilities of two models, mitigating excessive disparities to promote the utility of peer review. Extensive experiments demonstrate that CRM significantly enhances RM generalization, with up to 9.94 points improvement on RewardBench under an extreme 40\% noise. Moreover, CRM can seamlessly extend to implicit-reward alignment methods, offering a robust and versatile alignment strategy.

Figures (11)

• Figure 1: Simulated RL experiemnts with different proxy RMs. Policy optimized by clean RM shows an increasring gold score while noisy RM degrade the RL optimization.
• Figure 2: Characterizing robustness of preference instances. The training instances can be roughly classified into three types: robust, non-robust and ambiguous preferences.
• Figure 3: Win-rate comparison on Anthropic-Helpful and Harmless between policies optimized by RM trained on different group of preferences.
• Figure 4: The CRM framework: (1) Curriculum Learning organizes the preference learning process as a progression from easy to complex. (2) Peer Review enables the two RMs to collaborate by assessing each other’s data selections, filtering out potential noisy preferences within each batch.
• Figure 5: Left: Ablation study of different variants on HH-RLHF with varing noises. Middle: Performance of collaborative reward modeling with different-sized RMs. Right: Impact of $\lambda_t$ on preference accuracy across varying noisy-levels.