Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Guarantee of Reward Modeling Using Deep Neural Networks

Yuanhang Luo, Yeheng Ge, Ruijian Han, Guohao Shen

TL;DR

The paper develops a non-parametric theoretical framework for deep reward modeling using pairwise comparisons in RLHF. It introduces a margin-type condition on human preferences and derives architecture-dependent, non-asymptotic regret bounds that sharpen as the margin becomes clearer. By decomposing estimation error into stochastic and approximation components, the authors obtain concrete rates for deep reward estimators with width $W=O(d^{\beta})$ and depth $D=O(\sqrt{N})$, and show how data quality affects sample efficiency. Empirical results corroborate the theory, illustrating how network architecture balance and high-quality comparison data drive performance in Bradley–Terry and Thurstonian settings. Overall, the work provides a principled link between human-belief clarity, network design, and learning efficiency in RLHF reward modeling.

Abstract

In this work, we study the learning theory of reward modeling with pairwise comparison data using deep neural networks. We establish a novel non-asymptotic regret bound for deep reward estimators in a non-parametric setting, which depends explicitly on the network architecture. Furthermore, to underscore the critical importance of clear human beliefs, we introduce a margin-type condition that assumes the conditional winning probability of the optimal action in pairwise comparisons is significantly distanced from 1/2. This condition enables a sharper regret bound, which substantiates the empirical efficiency of Reinforcement Learning from Human Feedback and highlights clear human beliefs in its success. Notably, this improvement stems from high-quality pairwise comparison data implied by the margin-type condition, is independent of the specific estimators used, and thus applies to various learning algorithms and models.

Learning Guarantee of Reward Modeling Using Deep Neural Networks

TL;DR

The paper develops a non-parametric theoretical framework for deep reward modeling using pairwise comparisons in RLHF. It introduces a margin-type condition on human preferences and derives architecture-dependent, non-asymptotic regret bounds that sharpen as the margin becomes clearer. By decomposing estimation error into stochastic and approximation components, the authors obtain concrete rates for deep reward estimators with width and depth , and show how data quality affects sample efficiency. Empirical results corroborate the theory, illustrating how network architecture balance and high-quality comparison data drive performance in Bradley–Terry and Thurstonian settings. Overall, the work provides a principled link between human-belief clarity, network design, and learning efficiency in RLHF reward modeling.

Abstract

In this work, we study the learning theory of reward modeling with pairwise comparison data using deep neural networks. We establish a novel non-asymptotic regret bound for deep reward estimators in a non-parametric setting, which depends explicitly on the network architecture. Furthermore, to underscore the critical importance of clear human beliefs, we introduce a margin-type condition that assumes the conditional winning probability of the optimal action in pairwise comparisons is significantly distanced from 1/2. This condition enables a sharper regret bound, which substantiates the empirical efficiency of Reinforcement Learning from Human Feedback and highlights clear human beliefs in its success. Notably, this improvement stems from high-quality pairwise comparison data implied by the margin-type condition, is independent of the specific estimators used, and thus applies to various learning algorithms and models.
Paper Structure (23 sections, 10 theorems, 68 equations, 7 figures)

This paper contains 23 sections, 10 theorems, 68 equations, 7 figures.

Table of Contents

  1. Introduction
  2. Pairwise Comparison, Margin-type Condition and Sharper Bound
  3. Sharper Regret Bound
  4. Learning Guarantee of Deep Reward Modeling
  5. Estimation Within Deep Neural Network Function Class
  6. Experiments
  7. Experiment 1: Neural Network Configurations
  8. Experiment 2: High-quality Pairwise Comparison Dataset
  9. Related Works
  10. Conclusion
  11. Technical Notations
  12. Proof of Lemma \ref{['noiser']}
  13. Proof of Theorem \ref{['faster']}
  14. Proof of Corollary \ref{['without_TNC']}
  15. Discussion on Selection Consistency
  16. ...and 8 more sections

Key Result

Lemma 2.4

Given Assumption noisep, with $\alpha \in (0,1)$ and $t \in (0, c_{r^*})$ where $c_{r^*}$ is the upper bound of the true reward defined in Assumption dynamic_range, there exists a universal constant $c_g^\prime$ such that Specifically, $c^\prime_g= (1/4)^{\alpha/(1-\alpha)}c_g$ in the BT model and $c^\prime_g = (1/2\pi)^{\alpha/(2-2\alpha)}c_g$ in the Thurstonian model where $c_g$ is a universal

Figures (7)

  • Figure 1: Surface plot of regret for both BT (upper panel) and Thurstonian (lower panel) models for sinusoidal reward functions across different neural network configurations.
  • Figure 2: Empirical distribution of regret for both BT (upper panel) and Thurstonian (lower panel) models for sinusoidal reward functions under different noise levels. A larger noise level implies a lower quality of the comparison dataset, resulting in higher regret.
  • Figure 3: Surface plot of regret for both BT (left panel) and Thurstonian (right panel) models for Hermite-Gaussian reward functions across different neural network configurations.
  • Figure 4: Surface plot of regret for both BT (left panel) and Thurstonian (right panel) models for nonlinear composite sinusoid reward functions across different neural network configurations.
  • Figure 5: Empirical distribution of regret for both BT (left panel) and Thurstonian (right panel) models for Hermite-Gaussian reward functions under different noise levels.
  • ...and 2 more figures

Theorems & Definitions (19)

  • Example 2.2: Bradley-Terry (BT) model bradley1952rank
  • Example 2.3: Thurstonian model thurstone2017law
  • Lemma 2.4
  • Theorem 2.5: Faster Rate with Margin Condition
  • Corollary 2.6: Regret Bound without Margin Condition
  • Theorem 2.7: Informal, Guarantee for Deep Reward Modeling
  • Definition 3.1: Comparison Function
  • Lemma 3.4: Excess risk decomposition
  • Proposition 3.5: Stochastic Error Bound
  • Proposition 3.6: Approximation Error Bound
  • ...and 9 more