Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-Tuning Language Models with Reward Learning on Policy

Hao Lang, Fei Huang, Yongbin Li

TL;DR

Reward learning on policy (RLP) is proposed, an unsupervised framework that refines a reward model using policy samples to keep it on-distribution and an unsupervised multi-view learning method is introduced to learn robust representations of policy samples.

Abstract

Reinforcement learning from human feedback (RLHF) has emerged as an effective approach to aligning large language models (LLMs) to human preferences. RLHF contains three steps, i.e., human preference collecting, reward learning, and policy optimization, which are usually performed serially. Despite its popularity, however, (fixed) reward models may suffer from inaccurate off-distribution, since policy optimization continuously shifts LLMs' data distribution. Repeatedly collecting new preference data from the latest LLMs may alleviate this issue, which unfortunately makes the resulting system more complicated and difficult to optimize. In this paper, we propose reward learning on policy (RLP), an unsupervised framework that refines a reward model using policy samples to keep it on-distribution. Specifically, an unsupervised multi-view learning method is introduced to learn robust representations of policy samples. Meanwhile, a synthetic preference generation approach is developed to simulate high-quality preference data with policy outputs. Extensive experiments on three benchmark datasets show that RLP consistently outperforms the state-of-the-art. Our code is available at \url{https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/rlp}.

Fine-Tuning Language Models with Reward Learning on Policy

TL;DR

Reward learning on policy (RLP) is proposed, an unsupervised framework that refines a reward model using policy samples to keep it on-distribution and an unsupervised multi-view learning method is introduced to learn robust representations of policy samples.

Abstract

Reinforcement learning from human feedback (RLHF) has emerged as an effective approach to aligning large language models (LLMs) to human preferences. RLHF contains three steps, i.e., human preference collecting, reward learning, and policy optimization, which are usually performed serially. Despite its popularity, however, (fixed) reward models may suffer from inaccurate off-distribution, since policy optimization continuously shifts LLMs' data distribution. Repeatedly collecting new preference data from the latest LLMs may alleviate this issue, which unfortunately makes the resulting system more complicated and difficult to optimize. In this paper, we propose reward learning on policy (RLP), an unsupervised framework that refines a reward model using policy samples to keep it on-distribution. Specifically, an unsupervised multi-view learning method is introduced to learn robust representations of policy samples. Meanwhile, a synthetic preference generation approach is developed to simulate high-quality preference data with policy outputs. Extensive experiments on three benchmark datasets show that RLP consistently outperforms the state-of-the-art. Our code is available at \url{https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/rlp}.
Paper Structure (27 sections, 5 equations, 2 figures, 8 tables, 1 algorithm)

This paper contains 27 sections, 5 equations, 2 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Step 0, SFT:
  5. Step 1, Human preference collecting:
  6. Step 2, Reward learning:
  7. Step 3, RL policy optimization:
  8. Reward Learning on Policy
  9. Overview
  10. Reward Retraining
  11. Unsupervised Multi-View Learning
  12. Synthetic Preference Generation
  13. Policy Retraining
  14. Experiments
  15. Datasets
  16. ...and 12 more sections

Figures (2)

  • Figure 1: Comparison of standard RLHF (top) and RLHF with reward learning on policies (bottom). Different from (top), which performs reward learning and policy optimization serially, we iteratively train one of the two models with the help of the other.
  • Figure 2: The simulated win-rate (%) performance of RLP-SPG compared to PPO on various subsets of AlpacaFarm. Win-rates are computed against reference model text-davinci-003.