Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Filtered Direct Preference Optimization

Tetsuro Morimura, Mitsuki Sakamoto, Yuu Jinnai, Kenshi Abe, Kaito Ariu

TL;DR

The paper investigates how preference-text quality affects direct preference optimization (DPO) compared with reward-model-based RLHF. It reveals that DPO is more sensitive to data quality and proposes filtered Direct Preference Optimization (fDPO), which filters low-quality samples using a trained reward model before performing DPO. Across AlpacaFarm and Anthropic HH datasets, fDPO improves final model performance and robustness to mixed-quality data, demonstrating practical data-refinement as a path to better offline alignment. The approach balances data efficiency with quality control, offering a scalable solution when high-quality human data are scarce, and it is validated with both automated and human evaluations. Overall, fDPO advances offline alignment by mitigating dataset quality bottlenecks in RM-free learning while remaining compatible with standard RLHF evaluations.

Abstract

Reinforcement learning from human feedback (RLHF) plays a crucial role in aligning language models with human preferences. While the significance of dataset quality is generally recognized, explicit investigations into its impact within the RLHF framework, to our knowledge, have been limited. This paper addresses the issue of text quality within the preference dataset by focusing on direct preference optimization (DPO), an increasingly adopted reward-model-free RLHF method. We confirm that text quality significantly influences the performance of models optimized with DPO more than those optimized with reward-model-based RLHF. Building on this new insight, we propose an extension of DPO, termed filtered direct preference optimization (fDPO). fDPO uses a trained reward model to monitor the quality of texts within the preference dataset during DPO training. Samples of lower quality are discarded based on comparisons with texts generated by the model being optimized, resulting in a more accurate dataset. Experimental results demonstrate that fDPO enhances the final model performance. Our code is available at https://github.com/CyberAgentAILab/filtered-dpo.

Filtered Direct Preference Optimization

TL;DR

The paper investigates how preference-text quality affects direct preference optimization (DPO) compared with reward-model-based RLHF. It reveals that DPO is more sensitive to data quality and proposes filtered Direct Preference Optimization (fDPO), which filters low-quality samples using a trained reward model before performing DPO. Across AlpacaFarm and Anthropic HH datasets, fDPO improves final model performance and robustness to mixed-quality data, demonstrating practical data-refinement as a path to better offline alignment. The approach balances data efficiency with quality control, offering a scalable solution when high-quality human data are scarce, and it is validated with both automated and human evaluations. Overall, fDPO advances offline alignment by mitigating dataset quality bottlenecks in RM-free learning while remaining compatible with standard RLHF evaluations.

Abstract

Reinforcement learning from human feedback (RLHF) plays a crucial role in aligning language models with human preferences. While the significance of dataset quality is generally recognized, explicit investigations into its impact within the RLHF framework, to our knowledge, have been limited. This paper addresses the issue of text quality within the preference dataset by focusing on direct preference optimization (DPO), an increasingly adopted reward-model-free RLHF method. We confirm that text quality significantly influences the performance of models optimized with DPO more than those optimized with reward-model-based RLHF. Building on this new insight, we propose an extension of DPO, termed filtered direct preference optimization (fDPO). fDPO uses a trained reward model to monitor the quality of texts within the preference dataset during DPO training. Samples of lower quality are discarded based on comparisons with texts generated by the model being optimized, resulting in a more accurate dataset. Experimental results demonstrate that fDPO enhances the final model performance. Our code is available at https://github.com/CyberAgentAILab/filtered-dpo.
Paper Structure (44 sections, 1 theorem, 9 equations, 8 figures, 6 tables)

This paper contains 44 sections, 1 theorem, 9 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Background
  4. Reinforcement Learning from Human Feedback
  5. Supervised fine-tuning.
  6. Reward Modelling.
  7. RL fine-tuning.
  8. Direct Preference Optimization
  9. Filtered Direct Preference Optimization
  10. fDPO Implementation
  11. Background and Motivation for fDPO
  12. Experiments
  13. Pretrained Models
  14. Datasets
  15. Low-quality dataset.
  16. ...and 29 more sections

Key Result

Proposition B.1

Let the following assumptions hold: When the DPO algorithm updates the parameter $\theta$ with where $\alpha$ is the learning rate and is sufficiently small, the sensitivity of $\pi_\theta(y_c)$, defined as the magnitude of change in probability, is approximately $\delta$ times higher than that of $\pi_\theta(y_r)$.

Figures (8)

  • Figure 1: Performance comparison of alignment methods using a 160M LM with the AlpacaFarm dataset AlpacaFarm23, where the gold rewards are adjusted so that the average reward of the initial LM is zero. (A) shows the impact of dataset quality on RLHF InstructGPT and DPO DPO, with DPO exhibiting greater sensitivity to dataset quality variations. (B) compares the performance of DPO and the proposed fDPO on a mixed-quality dataset, illustrating that fDPO effectively mitigates the impact of data quality variations.
  • Figure 2: Performance comparison between DPO and fDPO using a 1.4B-sized LM on the mix-quality dataset.
  • Figure 3:
  • Figure 6: Results of the empirical validation of the assumptions in Proposition \ref{['prop:sensitivity']}. The plots display the log delta $\log(\pi(y_i|x)/\pi(y_j|x))$, log norm ratio $\log(\|\nabla\log\pi(y_i|x) \| / \| \nabla\log\pi(y_j|x) \| )$, and cosine similarity $\cos(\nabla\log\pi(y_i|x), \nabla\log\pi(y_j|x))$ for 120 pairs of responses generated by GPT-2 large. The results show that the assumptions largely hold, with minimal dependence on $\delta=\pi(y_i|x)/\pi(y_j|x)$.
  • Figure 7: The learning curves for DPO and fDPO using the 160M-sized LM on the mix-quality dataset of AlpacaFarm. The horizontal axes of the figures represent the number of training steps and the KL divergence with the initial LM (SFT model), respectively, where the gold rewards are adjusted so that the average reward of the SFT model is zero.
  • ...and 3 more figures

Theorems & Definitions (1)

  • Proposition B.1