Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ImplicitRM: Unbiased Reward Modeling from Implicit Preference Data for LLM alignment

Hao Wang, Haocheng Yang, Licheng Pan, Lei Shen, Xiaoxi Li, Yinuo Wang, Zhichao Chen, Yuan Lu, Haoxuan Li, Zhouchen Lin

Abstract

Reward modeling represents a long-standing challenge in reinforcement learning from human feedback (RLHF) for aligning language models. Current reward modeling is heavily contingent upon experimental feedback data with high collection costs. In this work, we study \textit{implicit reward modeling} -- learning reward models from implicit human feedback (e.g., clicks and copies) -- as a cost-effective alternative. We identify two fundamental challenges in implicit reward modeling: (1) Implicit preference data lacks definitive negative samples, which makes standard positive-negative classification methods inapplicable; (2) Implicit preference data suffers from user preference bias, where different responses have different propensities to elicit user feedback actions, which exacerbates the difficulty of distinguishing definitive negative samples. To address these challenges, we propose ImplicitRM, which aims to learn unbiased reward models from implicit preference data. ImplicitRM stratifies training samples into four latent groups via a stratification model. Building on this, it derives a learning objective through likelihood maximization, which we prove is theoretically unbiased, effectively resolving both challenges. Experiments demonstrate that ImplicitRM learns accurate reward models across implicit preference datasets. Code is available on our project website.

ImplicitRM: Unbiased Reward Modeling from Implicit Preference Data for LLM alignment

Abstract

Reward modeling represents a long-standing challenge in reinforcement learning from human feedback (RLHF) for aligning language models. Current reward modeling is heavily contingent upon experimental feedback data with high collection costs. In this work, we study \textit{implicit reward modeling} -- learning reward models from implicit human feedback (e.g., clicks and copies) -- as a cost-effective alternative. We identify two fundamental challenges in implicit reward modeling: (1) Implicit preference data lacks definitive negative samples, which makes standard positive-negative classification methods inapplicable; (2) Implicit preference data suffers from user preference bias, where different responses have different propensities to elicit user feedback actions, which exacerbates the difficulty of distinguishing definitive negative samples. To address these challenges, we propose ImplicitRM, which aims to learn unbiased reward models from implicit preference data. ImplicitRM stratifies training samples into four latent groups via a stratification model. Building on this, it derives a learning objective through likelihood maximization, which we prove is theoretically unbiased, effectively resolving both challenges. Experiments demonstrate that ImplicitRM learns accurate reward models across implicit preference datasets. Code is available on our project website.
Paper Structure (27 sections, 4 theorems, 13 equations, 4 figures, 6 tables, 1 algorithm)

This paper contains 27 sections, 4 theorems, 13 equations, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Reinforcement learning from human feedback
  4. Problem definition
  5. Methodology
  6. Motivation
  7. Stratification of implicit preference samples
  8. Maximum likelihood, evidence lower-bound and learning objective
  9. The workflow of ImplicitRM
  10. Experiments
  11. Experimental setup
  12. Overall performance
  13. Ablation study
  14. Generalization analysis
  15. Hyperparameter sensitivity analysis
  16. ...and 12 more sections

Key Result

Theorem 3.1

The evidence lower bound of the marginal log-likelihood $\mathcal{L}$ can be expressed as:

Figures (4)

  • Figure 1: Four typical generation processes of implicit preference data, where "copy" represents user feedback. We denote the true user preference as positive ($r^*=1$) or negative ($r^*=0$), and the user interaction as active ($a=1$) or passive ($a=0$). Different colors indicate different $r^*$. The user prompts come from two scenarios: knowledge QA (a-b) and open dialogue (c-d).
  • Figure 2: Performance comparison under different update rate $\eta$ and batch size $\mathrm{B}$ on HelpSteer.
  • Figure 3: Performance comparison under different proportions of positive samples that elicit user actions ($\alpha$).
  • Figure 4: The data generation process for implicit preference data. Gray nodes indicate unobservable (latent) variables.

Theorems & Definitions (8)

  • Theorem 3.1
  • proof
  • Theorem 3.2: Unbiasedness
  • proof
  • Theorem A.1
  • proof
  • Theorem A.2: Unbiasedness
  • proof