Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm

Sarvesh Shashidhar, Ritik, Nachiketa Patil, Suraj Racha, Ganesh Ramakrishnan

TL;DR

The paper tackles the robustness of segment-level preferences in 2D-DPO for aligning LLMs. It extends 2D-DPO with two noise-robust formulations: a high-level unbiased loss that handles random preference flips and a segment-level perturbation scheme that perturbs per-segment scores, both supported by theoretical arguments and empirical validation. Experiments on HelpSteer-2D with a 6.9B parameter model show that vanilla 2D-DPO degrades under segment-level noise, while the proposed Robust 2D-DPO maintains and even improves performance under noise, underscoring the practicality of granular, noise-aware preference optimization. These contributions advance robust, fine-grained alignment methods for LLMs in real-world noisy labeling settings.

Abstract

Direct Preference Optimisation (DPO) has emerged as a powerful method for aligning Large Language Models (LLMs) with human preferences, offering a stable and efficient alternative to approaches that use Reinforcement learning via Human Feedback. In this work, we investigate the performance of DPO using open-source preference datasets. One of the major drawbacks of DPO is that it doesn't induce granular scoring and treats all the segments of the responses with equal propensity. However, this is not practically true for human preferences since even "good" responses have segments that may not be preferred by the annotator. To resolve this, a 2-dimensional scoring for DPO alignment called 2D-DPO was proposed. We explore the 2D-DPO alignment paradigm and the advantages it provides over the standard DPO by comparing their win rates. It is observed that these methods, even though effective, are not robust to label/score noise. To counter this, we propose an approach of incorporating segment-level score noise robustness to the 2D-DPO algorithm. Along with theoretical backing, we also provide empirical verification in favour of the algorithm and introduce other noise models that can be present.

Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm

TL;DR

The paper tackles the robustness of segment-level preferences in 2D-DPO for aligning LLMs. It extends 2D-DPO with two noise-robust formulations: a high-level unbiased loss that handles random preference flips and a segment-level perturbation scheme that perturbs per-segment scores, both supported by theoretical arguments and empirical validation. Experiments on HelpSteer-2D with a 6.9B parameter model show that vanilla 2D-DPO degrades under segment-level noise, while the proposed Robust 2D-DPO maintains and even improves performance under noise, underscoring the practicality of granular, noise-aware preference optimization. These contributions advance robust, fine-grained alignment methods for LLMs in real-world noisy labeling settings.

Abstract

Direct Preference Optimisation (DPO) has emerged as a powerful method for aligning Large Language Models (LLMs) with human preferences, offering a stable and efficient alternative to approaches that use Reinforcement learning via Human Feedback. In this work, we investigate the performance of DPO using open-source preference datasets. One of the major drawbacks of DPO is that it doesn't induce granular scoring and treats all the segments of the responses with equal propensity. However, this is not practically true for human preferences since even "good" responses have segments that may not be preferred by the annotator. To resolve this, a 2-dimensional scoring for DPO alignment called 2D-DPO was proposed. We explore the 2D-DPO alignment paradigm and the advantages it provides over the standard DPO by comparing their win rates. It is observed that these methods, even though effective, are not robust to label/score noise. To counter this, we propose an approach of incorporating segment-level score noise robustness to the 2D-DPO algorithm. Along with theoretical backing, we also provide empirical verification in favour of the algorithm and introduce other noise models that can be present.
Paper Structure (20 sections, 1 theorem, 46 equations, 5 figures, 3 tables, 1 algorithm)

This paper contains 20 sections, 1 theorem, 46 equations, 5 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Setup and Background
  4. Methodology
  5. Understanding 2D-DPO
  6. High-Level Noise-Robust 2D-DPO Formulation
  7. Segment-Level Noise-Robust 2D-DPO Formulation
  8. Experiments
  9. Experiment 1
  10. Experiment 2
  11. Experiment 3
  12. Experiment 4
  13. Inferences of Experiments
  14. Conclusion and Future Work
  15. Proof for Direct Preference Optimisation
  16. ...and 5 more sections

Key Result

Lemma 1

For any $x \in \mathbb{R}$ and the $\sigma(x)$ begin defined as the standard Sigmoid function, we have -

Figures (5)

  • Figure 1: Win rate progression for Vanilla DPO
  • Figure 2: Win rate progression for Vanilla 2D-DPO (Noiseless)
  • Figure 3: Win rate progression for Vanilla 2D-DPO
  • Figure 4: Win rate progression for Robust 2D-DPO
  • Figure 5: Win rate vs. Sampling Temperature

Theorems & Definitions (1)

  • Lemma 1