Reward Modeling from Natural Language Human Feedback

TL;DR

This work identifies a fundamental mismatch between outcome-only supervision and the desired process-level reasoning in generative reward models (GRMs). It introduces Reward Modeling from Natural Language Human Feedback (RM-NLHF), which uses core similarities between human critiques and GRM critiques as a process reward, and complements outcome rewards to improve critique quality and preference accuracy. To scale process supervision, it proposes MetaRM, a meta reward model trained on critiques and capable of predicting process reward for data lacking critiques, and Online MetaRM, which continuously adapts during RL training. Across multiple benchmarks, RM-NLHF outperforms outcome-only GRMs, with online MetaRM achieving comparable gains to full human critique supervision while dramatically reducing annotation costs, and analysis shows richer, more diverse critiques than traditional outcome-focused feedback.

Abstract

Reinforcement Learning with Verifiable reward (RLVR) on preference data has become the mainstream approach for training Generative Reward Models (GRMs). Typically in pairwise rewarding tasks, GRMs generate reasoning chains ending with critiques and preference labels, and RLVR then relies on the correctness of the preference labels as the training reward. However, in this paper, we demonstrate that such binary classification tasks make GRMs susceptible to guessing correct outcomes without sound critiques. Consequently, these spurious successes introduce substantial noise into the reward signal, thereby impairing the effectiveness of reinforcement learning. To address this issue, we propose Reward Modeling from Natural Language Human Feedback (RM-NLHF), which leverages natural language feedback to obtain process reward signals, thereby mitigating the problem of limited solution space inherent in binary tasks. Specifically, we compute the similarity between GRM-generated and human critiques as the training reward, which provides more accurate reward signals than outcome-only supervision. Additionally, considering that human critiques are difficult to scale up, we introduce Meta Reward Model (MetaRM) which learns to predict process reward from datasets with human critiques and then generalizes to data without human critiques. Experiments on multiple benchmarks demonstrate that our method consistently outperforms state-of-the-art GRMs trained with outcome-only reward, confirming the superiority of integrating natural language over binary human feedback as supervision.

Figures (13)

• Figure 1: Examples of GRMs achieving correct outcome with flawed critiques.
• Figure 2: Inconsistency between the correctness of outcome and process across various models and tasks.
• Figure 3: Comparison between using human critique as process reward and outcome-only reward.
• Figure 4: Workflow of online MetaRM framework. The formal algorithm is summarized in Algorithm \ref{['alg:online_metarm']}.
• Figure 5: Comparison of offline and online MetaRM variants across multiple benchmarks.