Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prior Constraints-based Reward Model Training for Aligning Large Language Models

Hang Zhou, Chenglong Wang, Yimin Hu, Tong Xiao, Chunliang Zhang, Jingbo Zhu

TL;DR

Experimental results demonstrate that PCRM significantly improves alignment performance by effec-tively constraining reward score scaling, and is easily integrated intorary rank-based alignment methods, such as direct preference optimization, and can yield consistent improvement.

Abstract

Reinforcement learning with human feedback for aligning large language models (LLMs) trains a reward model typically using ranking loss with comparison pairs.However, the training procedure suffers from an inherent problem: the uncontrolled scaling of reward scores during reinforcement learning due to the lack of constraints while training the reward model.This paper proposes a Prior Constraints-based Reward Model (namely PCRM) training method to mitigate this problem. PCRM incorporates prior constraints, specifically, length ratio and cosine similarity between outputs of each comparison pair, during reward model training to regulate optimization magnitude and control score margins. We comprehensively evaluate PCRM by examining its rank correlation with human preferences and its effectiveness in aligning LLMs via RL. Experimental results demonstrate that PCRM significantly improves alignment performance by effectively constraining reward score scaling. As another bonus, our method is easily integrated into arbitrary rank-based alignment methods, such as direct preference optimization, and can yield consistent improvement.

Prior Constraints-based Reward Model Training for Aligning Large Language Models

TL;DR

Experimental results demonstrate that PCRM significantly improves alignment performance by effec-tively constraining reward score scaling, and is easily integrated intorary rank-based alignment methods, such as direct preference optimization, and can yield consistent improvement.

Abstract

Reinforcement learning with human feedback for aligning large language models (LLMs) trains a reward model typically using ranking loss with comparison pairs.However, the training procedure suffers from an inherent problem: the uncontrolled scaling of reward scores during reinforcement learning due to the lack of constraints while training the reward model.This paper proposes a Prior Constraints-based Reward Model (namely PCRM) training method to mitigate this problem. PCRM incorporates prior constraints, specifically, length ratio and cosine similarity between outputs of each comparison pair, during reward model training to regulate optimization magnitude and control score margins. We comprehensively evaluate PCRM by examining its rank correlation with human preferences and its effectiveness in aligning LLMs via RL. Experimental results demonstrate that PCRM significantly improves alignment performance by effectively constraining reward score scaling. As another bonus, our method is easily integrated into arbitrary rank-based alignment methods, such as direct preference optimization, and can yield consistent improvement.
Paper Structure (22 sections, 16 equations, 3 figures, 5 tables)

This paper contains 22 sections, 16 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Collecting Preference Data
  5. Training Reward Model
  6. RL Training Against the Reward Model
  7. Prior Constraints-based Reward Model
  8. Optimization Objective of PCRM
  9. Analysis of the Optimization for PCRM
  10. Experiments
  11. Datasets
  12. Settings
  13. Evaluation Metrics
  14. Results of Training Reward Models
  15. Results of Dialogue
  16. ...and 7 more sections

Figures (3)

  • Figure 1: The distribution between the margin of predicted reward scores and the similarity of paired data (calculated by cosine similarity or length ratio) with or without constraint on the dialogue task. Each green point corresponds to a single data sample. The x-axis refers to the similarity calculated by the cosine similarity of sentence embedding or by the ratio of sentence length. The y-axis refers to the margin of the reward scores.
  • Figure 2: PandaLM scores for different sampling temperatures using different methods. For each dialogue model, we conduct the generation three times and report the mean score of these generated responses.
  • Figure 3: The distribution between the margin of predicted reward scores and the similarity of paired data (calculated by cosine similarity or length ratio) with or without constraint on the summarization task.