Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Offline Preference Optimization via Maximum Marginal Likelihood Estimation

Saeed Najafi, Alona Fyshe

TL;DR

The paper addresses instability in RLHF-based alignment by introducing MMPO, an offline preference optimization method grounded in Maximum Marginal Likelihood. MMPO recasts alignment as maximizing the marginal likelihood of a preferred output using two-sample scores and a numerically stable log-sum-exp gradient, thereby achieving implicit preference optimization without an explicit reward function or entropy term. Theoretical analysis connects MMPO's gradient to preference amplification via a sigmoid of the score difference, and extensive experiments across 135M–8B parameter models show MMPO is stable to the hyperparameter $β$ and better preserves general language capabilities while delivering competitive AlpacaEval-2 win-rates. Ablation studies confirm the core mechanism, and the results suggest MMPO as a simpler, robust alternative for offline preference alignment with potential for online extension and broader applicability.

Abstract

Aligning Large Language Models (LLMs) with human preferences is crucial, but standard methods like Reinforcement Learning from Human Feedback (RLHF) are often complex and unstable. In this work, we propose a new, simpler approach that recasts alignment through the lens of Maximum Marginal Likelihood (MML) estimation. Our new MML based Preference Optimization (MMPO) maximizes the marginal log-likelihood of a preferred text output, using the preference pair as samples for approximation, and forgoes the need for both an explicit reward model and entropy maximization. We theoretically demonstrate that MMPO implicitly performs preference optimization, producing a weighted gradient that naturally up-weights chosen responses over rejected ones. Across models ranging from 135M to 8B parameters, we empirically show that MMPO: 1) is more stable with respect to the hyperparameter $β$ compared to alternative baselines, and 2) achieves competitive or superior preference alignment while better preserving the base model's general language capabilities. Through a series of ablation experiments, we show that this improved performance is indeed attributable to MMPO's implicit preference optimization within the gradient updates.

Offline Preference Optimization via Maximum Marginal Likelihood Estimation

TL;DR

The paper addresses instability in RLHF-based alignment by introducing MMPO, an offline preference optimization method grounded in Maximum Marginal Likelihood. MMPO recasts alignment as maximizing the marginal likelihood of a preferred output using two-sample scores and a numerically stable log-sum-exp gradient, thereby achieving implicit preference optimization without an explicit reward function or entropy term. Theoretical analysis connects MMPO's gradient to preference amplification via a sigmoid of the score difference, and extensive experiments across 135M–8B parameter models show MMPO is stable to the hyperparameter and better preserves general language capabilities while delivering competitive AlpacaEval-2 win-rates. Ablation studies confirm the core mechanism, and the results suggest MMPO as a simpler, robust alternative for offline preference alignment with potential for online extension and broader applicability.

Abstract

Aligning Large Language Models (LLMs) with human preferences is crucial, but standard methods like Reinforcement Learning from Human Feedback (RLHF) are often complex and unstable. In this work, we propose a new, simpler approach that recasts alignment through the lens of Maximum Marginal Likelihood (MML) estimation. Our new MML based Preference Optimization (MMPO) maximizes the marginal log-likelihood of a preferred text output, using the preference pair as samples for approximation, and forgoes the need for both an explicit reward model and entropy maximization. We theoretically demonstrate that MMPO implicitly performs preference optimization, producing a weighted gradient that naturally up-weights chosen responses over rejected ones. Across models ranging from 135M to 8B parameters, we empirically show that MMPO: 1) is more stable with respect to the hyperparameter compared to alternative baselines, and 2) achieves competitive or superior preference alignment while better preserving the base model's general language capabilities. Through a series of ablation experiments, we show that this improved performance is indeed attributable to MMPO's implicit preference optimization within the gradient updates.
Paper Structure (22 sections, 2 theorems, 17 equations, 3 figures, 1 table)

This paper contains 22 sections, 2 theorems, 17 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. Maximum Marginal Likelihood
  4. MML with Automatic Differentiation
  5. MML & Preference Optimization
  6. MML-based Preference Optimization (MMPO)
  7. Experiments
  8. Setup
  9. Results
  10. Language Evaluation
  11. Preference Evaluation
  12. Ablation Experiments
  13. Related Works
  14. Conclusion
  15. Limitations
  16. ...and 7 more sections

Key Result

Lemma 3.1

Let $s_1, s_2, \dots, s_n$ be real numbers, and let $s^* = \max_{i} s_i$. Then, $s_i - s^* \le 0$ and we have:

Figures (3)

  • Figure 1: The performance trade-off between preference alignment (AlpacaEval2 length-controlled win-rate) and general language capabilities (average LM Harness accuracy). The plots compare MMPO and DPO across four model sizes, with arrows indicating the performance trajectory over five training epochs for various $\beta$ values. Optimal performance would be for the model to retain general language capabilities while increasing preference optimization performance, indicated by lines moving towards the right while falling as little as possible along the y-axis. Stability across $\beta s$ is achieved when all $\beta$-specific trajectories for a particular model family are clustered together on the graph.
  • Figure 2: The performance trade-off between preference alignment (AlpacaEval2 length-controlled win-rate) and general language capabilities (average LM Harness accuracy). The plots compare MMPO and SimPO across four model sizes, with arrows indicating the performance trajectory over five training epochs for various $\beta$ values.
  • Figure 3: An ablation study of the MMPO objective ($\beta=0.05$) on the SmolLM2-360M model. The plot evaluates the trade-off between preference alignment, measured by the AlpacaEval2 length-controlled (LC) win-rate, and general language capabilities, measured by the average LM Harness accuracy. The tested modifications include removing the DPO-style auxiliary loss, removing in-batch normalization, adding length normalization, and adding entropy maximization. Arrows indicate the performance trajectory over five training epochs.

Theorems & Definitions (2)

  • Lemma 3.1
  • Theorem 3.2