Understanding Likelihood Over-optimisation in Direct Alignment Algorithms

TL;DR

This work explores the relationship between completion likelihood and model performance in state-of-the-art DAAs, and identifies a critical issue of likelihood over-optimisation, and identifies two key indicators that signal when over-optimised output diversity begins to harm performance.

Abstract

Direct Alignment Algorithms (DAAs), such as Direct Preference Optimisation (DPO) and Identity Preference Optimisation (IPO), have emerged as alternatives to online Reinforcement Learning from Human Feedback (RLHF) algorithms such as Proximal Policy Optimisation (PPO) for aligning language models to human preferences, without the need for explicit reward modelling. These methods generally aim to increase the likelihood of generating better (preferred) completions while discouraging worse (non-preferred) ones, while staying close to the original model's behaviour. In this work, we explore the relationship between completion likelihood and model performance in state-of-the-art DAAs, and identify a critical issue of likelihood over-optimisation. Contrary to expectations, we find that higher likelihood of better completions and larger margins between better and worse completion likelihoods do not necessarily lead to better performance, and may even degrade it. Our analysis reveals that while higher likelihood correlates with better memorisation of factual knowledge patterns, a slightly lower completion likelihood tends to improve output diversity, thus leading to better generalisation to unseen scenarios. Moreover, we identify two key indicators that signal when over-optimised output diversity begins to harm performance: Decreasing Entropy over Top-k Tokens and Diminishing Top-k Probability Mass. Our experimental results validate that these indicators are reliable signs of declining performance under different regularisations, helping prevent over-optimisation and improve alignment with human preferences.

Figures (12)

• Figure 1: Mean Log Likelihood (LLH) of Better Completion vs Win Probability (Left) and Average Number of Tokens in Model Outputs (Right). We report 7B and 35B model results on the UltraFeedback dataset. Our results indicate that: (1) A higher likelihood for better completions does not necessarily translate to higher win probability; and (2) There is no obvious correlation between the average number of tokens in model outputs and the likelihood of better completions.
• Figure 2: Learning curves across training steps for various metrics. Results are reported for the 7B models using the Hinge, DPO, and IPO on the UltraFeedback dataset.
• Figure 3: Win Probability Heatmaps Across Better and Worse Mean Log-Likelihoods. Results are reported for both 7B and 35B models on UltraFeedback and BinarizedPref datasets. Best performance does not always occur at the Pareto frontier of high likelihood for better completions and low likelihood for worse completions.
• Figure 4: Learning curves for DPO with different weights ($\lambda$) of NLL loss. We report the performance with different values of $\beta$ and $\lambda$ on the UltraFeedback dataset. A reversed entropy trend trending for entropy is a strong indicator of diversity over-optimisation.
• Figure 5: NaturalQuestionsOpen and TriviaQA vs Better Mean LLH on the UltraFeedback dataset. A higher LLH tends to memorise the factuality knowledge better.