Multi-Agent Collaborative Reward Design for Enhancing Reasoning in Reinforcement Learning

TL;DR

The paper tackles the insufficiency of a single scalar reward in RLHF for multi-dimensional language model alignment. It introduces Collaborative Reward Modeling (CRM), a multi-agent framework where specialist evaluators produce interpretable signals that are centrally fused into a single training reward, enabling standard policy-gradient updates with stability and robustness. RewardBench, a dedicated benchmark, accompanies the method to evaluate multi-dimensional feedback across reasoning, factuality, and safety, with experiments showing improved reasoning accuracy and generalization on GSM8K and RewardBench while maintaining dialogue quality. The approach offers a modular, interpretable path to safer, more reliable RLHF, scalable through plug-in evaluators and adaptable reward aggregation.

Abstract

We present CRM (Multi-Agent Collaborative Reward Model), a framework that replaces a single black-box reward model with a coordinated team of specialist evaluators to improve robustness and interpretability in RLHF. Conventional reward models struggle to jointly optimize multiple, sometimes conflicting, preference dimensions (e.g., factuality, helpfulness, safety) and offer limited transparency into why a score is assigned. CRM addresses these issues by decomposing preference evaluation into domain-specific agents that each produce partial signals, alongside global evaluators such as ranker-based and embedding-similarity rewards. A centralized aggregator fuses these signals at each timestep, balancing factors like step-wise correctness, multi-agent agreement, and repetition penalties, yielding a single training reward compatible with standard RL pipelines. The policy is optimized with advantage-based updates (e.g., GAE), while a value model regresses to the aggregated reward, enabling multi-perspective reward shaping without requiring additional human annotations beyond those used to train the evaluators. To support training and assessment, we introduce rewardBench, a benchmark and training suite aligned with the collaborative structure of CRM. Together, CRM and rewardBench provide a practical, modular path to more transparent reward modeling and more stable optimization.

Figures (2)

• Figure 1: Architecture of RewardBench. In comparison with the predefined and fixed reward function in the conventional method, Rewardbench leverages a multi-agent system to build an extensible intelligent reward function.
• Figure 2: Decomposition of collaborative reward roles.