Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback

Ang Li, Qiugen Xiao, Peng Cao, Jian Tang, Yi Yuan, Zijie Zhao, Xiaoyuan Chen, Liang Zhang, Xiangyang Li, Kaitong Yang, Weidong Guo, Yukang Gan, Xu Yu, Daniell Wang, Ying Shan

TL;DR

Human evaluation results show that HRLAIF inherits the ability of RLAIF to enhance human preference for outcomes at a low cost while also improving the satisfaction rate of responses, effectively addressing the issue of a decrease in satisfaction rate after basic RLAIF.

Abstract

Reinforcement Learning from AI Feedback (RLAIF) has the advantages of shorter annotation cycles and lower costs over Reinforcement Learning from Human Feedback (RLHF), making it highly efficient during the rapid strategy iteration periods of large language model (LLM) training. Using ChatGPT as a labeler to provide feedback on open-domain prompts in RLAIF training, we observe an increase in human evaluators' preference win ratio for model responses, but a decrease in evaluators' satisfaction rate. Analysis suggests that the decrease in satisfaction rate is mainly due to some responses becoming less helpful, particularly in terms of correctness and truthfulness, highlighting practical limitations of basic RLAIF. In this paper, we propose Hybrid Reinforcement Learning from AI Feedback (HRLAIF). This method enhances the accuracy of AI annotations for responses, making the model's helpfulness more robust in training process. Additionally, it employs AI for Red Teaming, further improving the model's harmlessness. Human evaluation results show that HRLAIF inherits the ability of RLAIF to enhance human preference for outcomes at a low cost while also improving the satisfaction rate of responses. Compared to the policy model before Reinforcement Learning (RL), it achieves an increase of 2.08\% in satisfaction rate, effectively addressing the issue of a decrease of 4.58\% in satisfaction rate after basic RLAIF.

HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback

TL;DR

Human evaluation results show that HRLAIF inherits the ability of RLAIF to enhance human preference for outcomes at a low cost while also improving the satisfaction rate of responses, effectively addressing the issue of a decrease in satisfaction rate after basic RLAIF.

Abstract

Reinforcement Learning from AI Feedback (RLAIF) has the advantages of shorter annotation cycles and lower costs over Reinforcement Learning from Human Feedback (RLHF), making it highly efficient during the rapid strategy iteration periods of large language model (LLM) training. Using ChatGPT as a labeler to provide feedback on open-domain prompts in RLAIF training, we observe an increase in human evaluators' preference win ratio for model responses, but a decrease in evaluators' satisfaction rate. Analysis suggests that the decrease in satisfaction rate is mainly due to some responses becoming less helpful, particularly in terms of correctness and truthfulness, highlighting practical limitations of basic RLAIF. In this paper, we propose Hybrid Reinforcement Learning from AI Feedback (HRLAIF). This method enhances the accuracy of AI annotations for responses, making the model's helpfulness more robust in training process. Additionally, it employs AI for Red Teaming, further improving the model's harmlessness. Human evaluation results show that HRLAIF inherits the ability of RLAIF to enhance human preference for outcomes at a low cost while also improving the satisfaction rate of responses. Compared to the policy model before Reinforcement Learning (RL), it achieves an increase of 2.08\% in satisfaction rate, effectively addressing the issue of a decrease of 4.58\% in satisfaction rate after basic RLAIF.
Paper Structure (26 sections, 5 equations, 7 figures, 7 tables)

This paper contains 26 sections, 5 equations, 7 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Basic AI Preference Labeling
  5. Hybrid AI Preference Labeling
  6. Hybrid Helpfulness Labeling
  7. Hybrid Harmlessness Labeling
  8. RM Training and PPO
  9. Experiment Details
  10. Data
  11. Cost Analysis of Preference Labeling
  12. Preference Labeling Quality Check
  13. Pretraining and SFT
  14. Training Details
  15. Reward Curves
  16. ...and 11 more sections

Figures (7)

  • Figure 1: Human evaluation results of basic RLAIF and HRLAIF on different prompt categories.
  • Figure 2: Framework of basic RLAIF and HRLAIF.
  • Figure 3: Train Data Proportioning
  • Figure 4: Reward Curves in RL. (a) and (d) display the tendency of the rewards printed during the training process (i.e., the mean reward score of prompts and responses trained in each PPO step). (b) and (e) show the tendency of rewards for each category on the subset of training set. (c) and (f) show the tendency of rewards on the test set.
  • Figure 5: Benchmark for helpfulness results. The horizontal axis represents the number of training steps, and the vertical axis represents the benchmark score. The red dashed line in the graph represents the score of $M_{SFT}$ before RL.
  • ...and 2 more figures