Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Differences Between Direct Alignment Algorithms are a Blur

Alexey Gorbatovski, Boris Shaposhnikov, Viacheslav Sinii, Alexey Malakhov, Daniil Gavrilov

TL;DR

This work provides a unified, theory-grounded comparison of Direct Alignment Algorithms (DAAs) for language model alignment, showing that explicit SFT stages and a tunable tempering parameter $β$ help harmonize odds-based methods with others. It demonstrates that the choice between pairwise and pointwise ranking is the dominant factor in alignment quality, particularly at larger model scales, due to how these objectives interact with prompt biases. The study also reveals that most DAAs reach near-optimal alignment with only 5–10% of SFT data, underscoring data efficiency and challenging claims of one-method superiority. Collectively, the findings advocate for nuanced, bias-aware evaluations and a two-stage, $eta$-tempered framework to fairly compare DAAs across tasks and model sizes.

Abstract

Direct Alignment Algorithms (DAAs) offer a simpler way to language model alignment than traditional RLHF by directly optimizing policies. While DAAs differ in their use of SFT (one-stage vs. two-stage), the scalar scores within their objectives (likelihood vs. odds ratios), and ranking objectives (pairwise vs. pointwise), the critical factors for performance remain underexplored. We provide a systematic comparative analysis. We first show that one-stage methods (e.g. ORPO, ASFT) underperform compared to two-stage approaches. However, we demonstrate that adapting them to a two-stage setup with an explicit SFT phase can improve their performance. Further, introducing and tuning a unifying $β$ parameter within this two-stage framework boosts their performence (e.g., AlpacaEval 2: $+13.45$ ORPO, $+8.27$ ASFT), matching established methods like DPO and enabling fair comparisons. Our comprehensive analysis reveals that the choice between pairwise and pointwise objectives is the primary determinant of alignment success, rather than the specific scalar score (e.g., policy-reference ratio vs. odds ratio) employed. We provide empirical evidence suggesting this stems from how these objectives interact with prompt-specific biases. These findings underscore the need for nuanced evaluations in DAA research to avoid oversimplified claims of superiority.

The Differences Between Direct Alignment Algorithms are a Blur

TL;DR

This work provides a unified, theory-grounded comparison of Direct Alignment Algorithms (DAAs) for language model alignment, showing that explicit SFT stages and a tunable tempering parameter help harmonize odds-based methods with others. It demonstrates that the choice between pairwise and pointwise ranking is the dominant factor in alignment quality, particularly at larger model scales, due to how these objectives interact with prompt biases. The study also reveals that most DAAs reach near-optimal alignment with only 5–10% of SFT data, underscoring data efficiency and challenging claims of one-method superiority. Collectively, the findings advocate for nuanced, bias-aware evaluations and a two-stage, -tempered framework to fairly compare DAAs across tasks and model sizes.

Abstract

Direct Alignment Algorithms (DAAs) offer a simpler way to language model alignment than traditional RLHF by directly optimizing policies. While DAAs differ in their use of SFT (one-stage vs. two-stage), the scalar scores within their objectives (likelihood vs. odds ratios), and ranking objectives (pairwise vs. pointwise), the critical factors for performance remain underexplored. We provide a systematic comparative analysis. We first show that one-stage methods (e.g. ORPO, ASFT) underperform compared to two-stage approaches. However, we demonstrate that adapting them to a two-stage setup with an explicit SFT phase can improve their performance. Further, introducing and tuning a unifying parameter within this two-stage framework boosts their performence (e.g., AlpacaEval 2: ORPO, ASFT), matching established methods like DPO and enabling fair comparisons. Our comprehensive analysis reveals that the choice between pairwise and pointwise objectives is the primary determinant of alignment success, rather than the specific scalar score (e.g., policy-reference ratio vs. odds ratio) employed. We provide empirical evidence suggesting this stems from how these objectives interact with prompt-specific biases. These findings underscore the need for nuanced evaluations in DAA research to avoid oversimplified claims of superiority.

Paper Structure

This paper contains 43 sections, 5 theorems, 23 equations, 18 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Modeling Sequences
  4. Reinforcement Learning from Human Feedback
  5. Direct Alignment Algorithms
  6. Single-Stage Alignment Methods
  7. Method
  8. Generalizing ASFT and ORPO
  9. ORPO and ASFT can operate without the SFT loss term and as two-stage methods
  10. Tempering ASFT and ORPO
  11. On the Difference Between Direct Alignment Algorithms
  12. Experimental Setup
  13. Base vs SFT-Initialized Models.
  14. $\beta$ Sensitivity.
  15. SFT Data Quantity.
  16. ...and 28 more sections

Key Result

Lemma B.1

Figures (18)

  • Figure 1: Overview of our work and main finding.Left: Existing DAA methods differ in use of SFT and $\beta$ parameter. Center: We unify methods by making SFT and $\beta$ explicit for each, showing that ORPO and ASFT can be brought into the same framework as other DAAs. Right: We compare DAAs along two axes (scalar score type and ranking type) and find that ranking type (pairwise, green vs. pointwise, red) is the main driver of alignment quality after unification.
  • Figure 2: Impact of the $\beta$ Parameter on ASFT and ORPO Alignment Quality. The plot shows how tuning $\beta$ (Section \ref{['sec:beta_to_one_stage']}) affects both ASFT and ORPO performance. Results are reported for GPT-4 Win Rate in the Llama 3.2 3B TL;DR setup and for AlpacaEval 2 LC Win Rate in the Llama 3.1 8B UF scenario. All other hyperparameters (e.g., learning rates) are selected via grid search, using each method's best configuration at $\beta = 1$ as the baseline. See Section \ref{['sec:res:beta']} for more details.
  • Figure 3: GPT-4 Evaluation of Llama 3.2 3B TL;DR setup. The comparison shows multiple alignment methods (rows) using their best hyperparameters, where each approach aims to generate concise and accurate summaries. Most methods exceed 90% Win Rate; ASFT achieves 87.2%, maintaining robust summarization performance. See Section \ref{['sec:res:pareto']} for more details.
  • Figure 4: Impact of SFT Dataset Size on Alignment Quality. Performance of the pairwise (a) and pointwise (b) alignment methods on AlpacaEval 2 (LC WR metric) when the SFT policy is trained on different fractions of the UltraChat dataset. Even a small fraction of SFT data (e.g., 5-10%) yields substantial gains over starting from the raw base model. See Section \ref{['sec:res:sft']} for more details.
  • Figure 8: Pareto front for alignment quality and KL divergence. Results for Llama 3.2 3B TL;DR and UF setups on GPT-4 Win Rate vs. "golden" validation subset and AlpacaEval 2 LC respectively with different $\beta$ values. Methods are grouped into pairwise and pointwise categories. For the summarization task (Llama 3.2 3B TL;DR), both pointwise and pairwise methods achieve strong overall results. For the UF setup, methods also perform similarly within overlapping confidence intervals, indicating no clear separation.
  • ...and 13 more figures

Theorems & Definitions (9)

  • Lemma B.1
  • proof
  • Lemma B.2
  • proof
  • Theorem B.3
  • proof
  • Theorem C.1
  • proof
  • Corollary C.2