Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks

Hao Wang, Licheng Pan, Zhichao Chen, Chunyuan Zheng, Zhixuan Chu, Xiaoxi Li, Yuan Lu, Xinggao Liu, Haoxuan Li, Zhouchen Lin

Abstract

Despite the success of reinforcement learning from human feedback (RLHF) in aligning language models, current reward modeling heavily relies on experimental feedback data collected from human annotators under controlled and costly conditions. In this work, we introduce observational reward modeling -- learning reward models with observational user feedback (e.g., clicks, copies, and upvotes) -- as a scalable and cost-effective alternative. We identify two fundamental challenges in this setting: (1) observational feedback is noisy due to annotation errors, which deviates it from true user preference; (2) observational feedback is biased by user preference, where users preferentially provide feedback on responses they feel strongly about, which creats a distribution shift between training and inference data. To address these challenges, we propose CausalRM, a causal-theoretic reward modeling framework that aims to learn unbiased reward models from observational feedback. To tackle challenge (1), CausalRM introduces a noise-aware surrogate loss term that is provably equivalent to the primal loss under noise-free conditions by explicitly modeling the annotation error generation process. To tackle challenge (2), CausalRM uses propensity scores -- the probability of a user providing feedback for a given response -- to reweight training samples, yielding a loss function that eliminates user preference bias. Extensive experiments across diverse LLM backbones and benchmark datasets validate that CausalRM effectively learns accurate reward signals from noisy and biased observational feedback and delivers substantial performance improvements on downstream RLHF tasks -- including a 49.2% gain on WildGuardMix and a 32.7% improvement on HarmBench. Code is available on our project website.

CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks

Abstract

Despite the success of reinforcement learning from human feedback (RLHF) in aligning language models, current reward modeling heavily relies on experimental feedback data collected from human annotators under controlled and costly conditions. In this work, we introduce observational reward modeling -- learning reward models with observational user feedback (e.g., clicks, copies, and upvotes) -- as a scalable and cost-effective alternative. We identify two fundamental challenges in this setting: (1) observational feedback is noisy due to annotation errors, which deviates it from true user preference; (2) observational feedback is biased by user preference, where users preferentially provide feedback on responses they feel strongly about, which creats a distribution shift between training and inference data. To address these challenges, we propose CausalRM, a causal-theoretic reward modeling framework that aims to learn unbiased reward models from observational feedback. To tackle challenge (1), CausalRM introduces a noise-aware surrogate loss term that is provably equivalent to the primal loss under noise-free conditions by explicitly modeling the annotation error generation process. To tackle challenge (2), CausalRM uses propensity scores -- the probability of a user providing feedback for a given response -- to reweight training samples, yielding a loss function that eliminates user preference bias. Extensive experiments across diverse LLM backbones and benchmark datasets validate that CausalRM effectively learns accurate reward signals from noisy and biased observational feedback and delivers substantial performance improvements on downstream RLHF tasks -- including a 49.2% gain on WildGuardMix and a 32.7% improvement on HarmBench. Code is available on our project website.
Paper Structure (21 sections, 3 theorems, 33 equations, 7 figures, 8 tables, 1 algorithm)

This paper contains 21 sections, 3 theorems, 33 equations, 7 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Reinforcement Learning from Human Feedback (RLHF)
  4. Problem Definition
  5. Methodology
  6. Motivation
  7. Noise-aware Surrogate Loss
  8. Causally Reweighted Learning Objective
  9. The Workflow of CausalRM
  10. Experiments
  11. Experimental Setup
  12. Overall Performance
  13. Ablation Study
  14. Synthetic Dataset Performance
  15. Hyperparameter Sensitivity
  16. ...and 6 more sections

Key Result

Theorem 1

The objective $\mathcal{L}_\mathrm{IPS}^*$ is an unbiased estimator of $\mathcal{L}_\mathrm{ideal}$ given accurate estimation of noise ratios ($\hat{\rho}_{01}=\rho_{01}$ and $\hat{\rho}_{10}=\rho_{10}$) and the propensity score is accurately estimated ($\hat{p}_\phi(x_i)=p_i$).

Figures (7)

  • Figure 1: The case study demonstrating user annotation errors in two typical scenarios. Different colors indicate different $r^*$.
  • Figure 2: The case study demonstrating user preference bias by comparing two typical scenarios in (a) and (b). Different colors indicate different $r^*$.
  • Figure 3: Performance comparison under different learning rate and batch size on PKU-SafeRLHF.
  • Figure 4: Performance comparison under different noise strength $\rho$ on three datasets: HelpSteer, UltraFeedback, PKU-SafeRLHF from left to right panels.
  • Figure 5: Performance comparison under different bias mildness $\alpha$ on three datasets: HelpSteer, UltraFeedback, PKU-SafeRLHF from left to right panels.
  • ...and 2 more figures

Theorems & Definitions (6)

  • Theorem 1: Unbiasedness of $\mathcal{L}_\mathrm{IPS}^*$
  • proof
  • Theorem 2: Unbiasedness of $\mathcal{L}_\mathrm{DR}^*$ & Double robustness
  • proof
  • Theorem 3: Variance reduction of $\mathcal{L}_\mathrm{DR}^*$
  • proof