Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Length-Controlled Margin-Based Preference Optimization without Reference Model

Gengxu Li, Tingyu Xia, Yi Chang, Yuan Wu

TL;DR

This work addresses offline preference optimization for large language models by tackling key drawbacks of Direct Preference Optimization (DPO), notably length bias, memory inefficiency, and probability degradation. It introduces Length-Controlled Margin-Based Preference Optimization (LMPO), a reference-free framework that uses average log-probability and a length-controlled margin-based loss within a Bradley-Terry style ranking, including a home-field extension and two normalization schemes. LMPO demonstrates competitive performance across Mistral-7B and Llama-3-8B on AlpacaEval 2 and Arena-Hard while achieving shorter prompts and reduced probability degradation, and it does so with lower memory requirements than DPO and SimPO. The paper provides extensive ablations, gradient analyses, and hyperparameter studies, underscoring LMPO’s practical viability for efficient, robust offline preference optimization in open-ended and instruction-following benchmarks, with open-source code available for reproducibility.

Abstract

Direct Preference Optimization (DPO) is a widely adopted offline algorithm for preference-based reinforcement learning from human feedback (RLHF), designed to improve training simplicity and stability by redefining reward functions. However, DPO is hindered by several limitations, including length bias, memory inefficiency, and probability degradation. To address these challenges, we propose Length-Controlled Margin-Based Preference Optimization (LMPO), a more efficient and robust alternative. LMPO introduces a uniform reference model as an upper bound for the DPO loss, enabling a more accurate approximation of the original optimization objective. Additionally, an average log-probability optimization strategy is employed to minimize discrepancies between training and inference phases. A key innovation of LMPO lies in its Length-Controlled Margin-Based loss function, integrated within the Bradley-Terry framework. This loss function regulates response length while simultaneously widening the margin between preferred and rejected outputs. By doing so, it mitigates probability degradation for both accepted and discarded responses, addressing a significant limitation of existing methods. We evaluate LMPO against state-of-the-art preference optimization techniques on two open-ended large language models, Mistral and LLaMA3, across six conditional benchmarks. Our experimental results demonstrate that LMPO effectively controls response length, reduces probability degradation, and outperforms existing approaches. The code is available at https://github.com/gengxuli/LMPO.

Length-Controlled Margin-Based Preference Optimization without Reference Model

TL;DR

This work addresses offline preference optimization for large language models by tackling key drawbacks of Direct Preference Optimization (DPO), notably length bias, memory inefficiency, and probability degradation. It introduces Length-Controlled Margin-Based Preference Optimization (LMPO), a reference-free framework that uses average log-probability and a length-controlled margin-based loss within a Bradley-Terry style ranking, including a home-field extension and two normalization schemes. LMPO demonstrates competitive performance across Mistral-7B and Llama-3-8B on AlpacaEval 2 and Arena-Hard while achieving shorter prompts and reduced probability degradation, and it does so with lower memory requirements than DPO and SimPO. The paper provides extensive ablations, gradient analyses, and hyperparameter studies, underscoring LMPO’s practical viability for efficient, robust offline preference optimization in open-ended and instruction-following benchmarks, with open-source code available for reproducibility.

Abstract

Direct Preference Optimization (DPO) is a widely adopted offline algorithm for preference-based reinforcement learning from human feedback (RLHF), designed to improve training simplicity and stability by redefining reward functions. However, DPO is hindered by several limitations, including length bias, memory inefficiency, and probability degradation. To address these challenges, we propose Length-Controlled Margin-Based Preference Optimization (LMPO), a more efficient and robust alternative. LMPO introduces a uniform reference model as an upper bound for the DPO loss, enabling a more accurate approximation of the original optimization objective. Additionally, an average log-probability optimization strategy is employed to minimize discrepancies between training and inference phases. A key innovation of LMPO lies in its Length-Controlled Margin-Based loss function, integrated within the Bradley-Terry framework. This loss function regulates response length while simultaneously widening the margin between preferred and rejected outputs. By doing so, it mitigates probability degradation for both accepted and discarded responses, addressing a significant limitation of existing methods. We evaluate LMPO against state-of-the-art preference optimization techniques on two open-ended large language models, Mistral and LLaMA3, across six conditional benchmarks. Our experimental results demonstrate that LMPO effectively controls response length, reduces probability degradation, and outperforms existing approaches. The code is available at https://github.com/gengxuli/LMPO.

Paper Structure

This paper contains 45 sections, 31 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Alignment with Reinforcement Learning
  4. Alignment Without Reward Models
  5. Alignment Without Reference Models
  6. Method
  7. Direct Preference Optimization (DPO)
  8. Improvement of DPO
  9. Bradley-Terry model with home-field advantage
  10. Removal of reference model
  11. Length-Controlled Margin-Based Loss
  12. Experiment
  13. Experimental Setup
  14. Main Results
  15. Analysis
  16. ...and 30 more sections

Figures (3)

  • Figure 1: Comparison with DPO and SimPO under the Mistral-Instruct and Llama3-Instruct models in the Arena-Hard benchmark. Our proposed method, LMPO, achieves the highest win rate while utilizing an exceptionally low average token count across both models.
  • Figure 2: The curves of the chosen (top) and rejected (bottom) log-probabilities during the training process in the Llama-3-Base (8B) setting. The blue and orange curves represent LMPO and SimPO, respectively.
  • Figure 3: The curves of the chosen log-probabilities during the training process in the Mistral-Base (7B) setting. The red, green and blue curves represent $\lambda$=0.05, $\lambda$=0.2 and $\lambda$=1.0, respectively.