Investigating on RLHF methodology

TL;DR

The approach for collecting a preference dataset through perplexity filtering is introduced, which makes the process of creating such a dataset for a specific Language Model much easier and more cost-effective.

Abstract

In this article, we investigate the alignment of Large Language Models according to human preferences. We discuss the features of training a Preference Model, which simulates human preferences, and the methods and details we found essential for achieving the best results. We also discuss using Reinforcement Learning to fine-tune Large Language Models and describe the challenges we faced and the ways to overcome them. Additionally, we present our experience with the Direct Preference Optimization method, which enables us to align a Large Language Model with human preferences without creating a separate Preference Model. As our contribution, we introduce the approach for collecting a preference dataset through perplexity filtering, which makes the process of creating such a dataset for a specific Language Model much easier and more cost-effective.

Figures (6)

• Figure 1: Average PM score for the responses generated by the trained LLM during the PPO training with both the plain average score and the average score minus the KLD-penalty for deviation from the basic LLM distribution. Additionally, the graph shows the average score for the responses from basic LLM for the same requests.
• Figure 2: Average length of response generated by the trained LLM during the PPO process.
• Figure 3: Side-by-side evaluation of the basic LLM (7b-SFT) versus the LLM trained with PPO (7b-PPO).
• Figure 4: Side-by-side evaluation of the basic LLM (7b-SFT) versus the LLM trained with DPO on our preference dataset (7b-DPO-pref-v1.0).
• Figure 5: Side-by-side comparison between the basic LLM (7b-SFT) and the LLM trained using DPO on the preference dataset that was created by filtering based on perplexity (7b-DPO-filtered).