RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models

TL;DR

RS-DPO addresses instability and high resource demands in PPO-based RLHF by marrying rejection sampling with direct preference optimization. It samples multiple responses directly from a supervised fine-tuned policy, ranks them via a learned reward model, and constructs synthetic preference data through PDGRS, which is then used in DPO to align the policy with human preferences. The approach yields stable performance across reward-model variants, reduces memory footprint by offline data generation, and outperforms RS, PPO, and DPO on MT-Bench and AlpacaEval benchmarks, particularly in limited-resource settings. This work offers a practical, efficient pathway for aligning LLMs without extensive on-policy sampling or large reward-model dependencies, with implications for scalable, accessible RLHF deployment.

Abstract

Reinforcement learning from human feedback (RLHF) has been extensively employed to align large language models with user intent. However, proximal policy optimization (PPO) based RLHF is occasionally unstable requiring significant hyperparameter finetuning, and computationally expensive to maximize the estimated reward during alignment. Recently, direct preference optimization (DPO) is proposed to address those challenges. However, DPO relies on contrastive responses generated from human annotator and alternative LLM, instead of the policy model, limiting the effectiveness of the RLHF. In this paper, we addresses both challenges by systematically combining rejection sampling (RS) and DPO. Our proposed method, RS-DPO, initiates with the development of a supervised fine-tuned policy model (SFT). A varied set of k responses per prompt are sampled directly from the SFT model. RS-DPO identifies pairs of contrastive samples based on their reward distribution. Finally, we apply DPO with the contrastive samples to align the model to human preference. Our experiments indicate that our proposed method effectively fine-tunes LLMs with limited resource environments, leading to improved alignment with user intent. Furthermore, it outperforms existing methods, including RS, PPO, and DPO.

Figures (4)

• Figure 1: The pipeline of RS-DPO. Our proposed method systematically combines rejection sampling (RS) and direct preference optimization (DPO). We start by creating a SFT model and use it to generate a diverse set of $k$ distinct responses for each prompt. Then, it selects a pair of contrastive samples based on their reward distribution. Subsequently, the method employs DPO to enhance the performance of the language model (LLM), thereby achieving improved alignment.
• Figure 2: Histograms of reward gap for WebGPT datasets with different reward models. The red dashed line represents the threshold value of 0.85 for preference data selection. Mean and standard deviation values of reward gaps are shown in the histograms.
• Figure 3: The left and right plots depict the reward accuracy and reward margin, respectively, of competing methods during DPO training on the Anthropic/HH-RLHF dataset.
• Figure 4: The left and right plots display the reward accuracy and reward margin, respectively, of competing methods during DPO training on the WebGPT dataset.