Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VRM: Teaching Reward Models to Understand Authentic Human Preferences

Biao Liu, Ning Xu, Junming Yang, Hao Xu, Xin Geng

TL;DR

VMM is proposed, i.e., Variational Reward Modeling, a novel framework that explicitly models the evaluation process of human preference judgments by incorporating both high-dimensional objective weights and low-dimensional semantic features as latent variables, which are inferred through variational inference techniques.

Abstract

Large Language Models (LLMs) have achieved remarkable success across diverse natural language tasks, yet the reward models employed for aligning LLMs often encounter challenges of reward hacking, where the approaches predominantly rely on directly mapping prompt-response pairs to scalar scores, which may inadvertently capture spurious correlations rather than authentic human preferences. In contrast, human evaluation employs a sophisticated process that initially weighs the relative importance of multiple high-dimensional objectives according to the prompt context, subsequently evaluating response quality through low-dimensional semantic features such as logical coherence and contextual appropriateness. Motivated by this consideration, we propose VRM, i.e., Variational Reward Modeling, a novel framework that explicitly models the evaluation process of human preference judgments by incorporating both high-dimensional objective weights and low-dimensional semantic features as latent variables, which are inferred through variational inference techniques. Additionally, we provide a theoretical analysis showing that VRM can achieve a tighter generalization error bound compared to the traditional reward model. Extensive experiments on benchmark datasets demonstrate that VRM significantly outperforms existing methods in capturing authentic human preferences.

VRM: Teaching Reward Models to Understand Authentic Human Preferences

TL;DR

VMM is proposed, i.e., Variational Reward Modeling, a novel framework that explicitly models the evaluation process of human preference judgments by incorporating both high-dimensional objective weights and low-dimensional semantic features as latent variables, which are inferred through variational inference techniques.

Abstract

Large Language Models (LLMs) have achieved remarkable success across diverse natural language tasks, yet the reward models employed for aligning LLMs often encounter challenges of reward hacking, where the approaches predominantly rely on directly mapping prompt-response pairs to scalar scores, which may inadvertently capture spurious correlations rather than authentic human preferences. In contrast, human evaluation employs a sophisticated process that initially weighs the relative importance of multiple high-dimensional objectives according to the prompt context, subsequently evaluating response quality through low-dimensional semantic features such as logical coherence and contextual appropriateness. Motivated by this consideration, we propose VRM, i.e., Variational Reward Modeling, a novel framework that explicitly models the evaluation process of human preference judgments by incorporating both high-dimensional objective weights and low-dimensional semantic features as latent variables, which are inferred through variational inference techniques. Additionally, we provide a theoretical analysis showing that VRM can achieve a tighter generalization error bound compared to the traditional reward model. Extensive experiments on benchmark datasets demonstrate that VRM significantly outperforms existing methods in capturing authentic human preferences.
Paper Structure (20 sections, 2 theorems, 33 equations, 6 figures, 2 tables)

This paper contains 20 sections, 2 theorems, 33 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Methodology
  5. Variational Reward Model
  6. Supervision of Objective Weights
  7. Theoretical Analysis
  8. A PAC-Bayes Generalization Bound
  9. Experiments
  10. Experimental Setup
  11. Main Results
  12. Further Analysis
  13. Comparison with Traditional Reward Model
  14. Loss Type of Supervision
  15. Parameter Sensitivity
  16. ...and 5 more sections

Key Result

Theorem 5.1

With probability at least $1-\delta$ over the draw of the training sample $\mathcal{S}$, the following holds: where $D(Q \| P)$ is the total KL divergence over the training set:

Figures (6)

  • Figure 1: Causal Graph of Vrm. The reward score $r$ is influenced by both high-dimensional objective weights $\bm{w}$ and low-dimensional semantic features $\bm{z}$.
  • Figure 2: Overview of the Vrm framework. The model processes prompt-response pairs through a shared backbone, generating multi-dimensional scores via the weight head and semantic features through the feature head. These components are combined with objective weights to produce the final reward predictions for preference learning.
  • Figure 3: Accuracy curves under different supervision loss types.
  • Figure 4: Accuracy curves comparing the traditional reward model (RM) and Vrm.
  • Figure 5: Parameter sensitivity analysis of the supervision loss weight $\lambda$.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Theorem 5.1: PAC-Bayes bound with decomposed KL
  • Lemma 2.1