Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ComPO: Preference Alignment via Comparison Oracles

Peter Chen, Xi Chen, Wotao Yin, Tianyi Lin

TL;DR

Direct preference alignment methods such as DPO can be verbose and susceptible to likelihood displacement when faced with noisy preference data. The authors propose ComPO, a zeroth-order, comparison-oracle–based approach that estimates sparse gradients from pairwise comparisons, with convergence guarantees for a smooth nonconvex objective and a practical scheme that confines perturbations to the output layer and uses gradient clipping. A three-step pipeline combines a clean-data DPO run with ComPO on noisy data (DPO_clean + ComPO), and experiments across Mistral-7B, Llama-3-8B, and Gemma-2-9B on AlpacaEval 2, Arena-Hard, and MT-Bench show improved length-controlled performance (LC) and often improved win rates (WR), validating the method’s ability to extract useful signal from noisy preferences. The results highlight the importance of specialized methods for preference pairs with distinct likelihood margins and demonstrate ComPO’s practical scalability and compatibility with existing direct alignment methods, offering a robust alternative for preference alignment with noisy data.

Abstract

Direct alignment methods are increasingly used for aligning large language models (LLMs) with human preferences. However, these methods suffer from the issues of verbosity and likelihood displacement, which can be driven by the noisy preference pairs that induce similar likelihood for preferred and dispreferred responses. The contributions of this paper are two-fold. First, we propose a new preference alignment method based on zeroth-order, comparison-based optimization via comparison oracles and provide convergence guarantees for its basic scheme. Second, we improve our method using some heuristics and conduct the experiments to demonstrate the flexibility and compatibility of practical scheme in improving the performance of LLMs using noisy preference pairs. Evaluations are conducted across multiple base and instruction-tuned models (Mistral-7B, Llama-3-8B and Gemma-2-9B) with benchmarks (AlpacaEval 2, MT-Bench and Arena-Hard). Experimental results show the effectiveness of our method as an alternative to addressing the limitations of existing direct alignment methods. A highlight of our work is that we evidence the importance of designing specialized methods for preference pairs with distinct likelihood margin, which complements the recent findings in Razin et al (2025).

ComPO: Preference Alignment via Comparison Oracles

TL;DR

Direct preference alignment methods such as DPO can be verbose and susceptible to likelihood displacement when faced with noisy preference data. The authors propose ComPO, a zeroth-order, comparison-oracle–based approach that estimates sparse gradients from pairwise comparisons, with convergence guarantees for a smooth nonconvex objective and a practical scheme that confines perturbations to the output layer and uses gradient clipping. A three-step pipeline combines a clean-data DPO run with ComPO on noisy data (DPO_clean + ComPO), and experiments across Mistral-7B, Llama-3-8B, and Gemma-2-9B on AlpacaEval 2, Arena-Hard, and MT-Bench show improved length-controlled performance (LC) and often improved win rates (WR), validating the method’s ability to extract useful signal from noisy preferences. The results highlight the importance of specialized methods for preference pairs with distinct likelihood margins and demonstrate ComPO’s practical scalability and compatibility with existing direct alignment methods, offering a robust alternative for preference alignment with noisy data.

Abstract

Direct alignment methods are increasingly used for aligning large language models (LLMs) with human preferences. However, these methods suffer from the issues of verbosity and likelihood displacement, which can be driven by the noisy preference pairs that induce similar likelihood for preferred and dispreferred responses. The contributions of this paper are two-fold. First, we propose a new preference alignment method based on zeroth-order, comparison-based optimization via comparison oracles and provide convergence guarantees for its basic scheme. Second, we improve our method using some heuristics and conduct the experiments to demonstrate the flexibility and compatibility of practical scheme in improving the performance of LLMs using noisy preference pairs. Evaluations are conducted across multiple base and instruction-tuned models (Mistral-7B, Llama-3-8B and Gemma-2-9B) with benchmarks (AlpacaEval 2, MT-Bench and Arena-Hard). Experimental results show the effectiveness of our method as an alternative to addressing the limitations of existing direct alignment methods. A highlight of our work is that we evidence the importance of designing specialized methods for preference pairs with distinct likelihood margin, which complements the recent findings in Razin et al (2025).
Paper Structure (25 sections, 4 theorems, 33 equations, 2 figures, 13 tables, 2 algorithms)

This paper contains 25 sections, 4 theorems, 33 equations, 2 figures, 13 tables, 2 algorithms.

Table of Contents

  1. Preliminaries and Technical Background
  2. Direct preference alignment
  3. Comparison oracles and zeroth-order methods
  4. Main Results
  5. Basic scheme with convergence guarantee
  6. Practical scheme
  7. Perturbations on output layer weights.
  8. Low-cost approximation to normalized sparse gradient estimation.
  9. Final method.
  10. Experiment
  11. Augmenting DPO and SimPO
  12. Ablation and scaling analysis
  13. Practical efficiency and compatibility
  14. Conclusion
  15. Limitations
  16. ...and 10 more sections

Key Result

Theorem 2.1

Suppose that there exists a smooth function $f$ satisfying (i) $f(\theta') < f(\theta)$ if and only if $\pi_{\theta'}(\mathbf y^+ | \mathbf x) > \pi_\theta(\mathbf y^+ | \mathbf x)$ and $\pi_{\theta'}(\mathbf y^- | \mathbf x) < \pi_\theta(\mathbf y^- | \mathbf x)$ for $\forall (\mathbf x, \mathbf y^ where $\ell > 0$ is the smoothness parameter of $f$ (i.e., $\|\nabla f(\theta) - \nabla f(\theta')\

Figures (2)

  • Figure 1: (Left) Percentage of non-zero entries in the final gradient across different gradient entry threshold $\lambda_g$; (Middle) Peak GPU memory usage across three models used in all experiments; (Right) Size of parameter space (output layer) in the comparison oracle perturbations and the run time for completing 600 perturbations using 30 NVIDIA A40 GPUs are shown.
  • Figure 2: Probability distribution and culmutive distribution of $m$ across noisy pairs. Dashed line shows the threshold used in Mistral-Base-7B.

Theorems & Definitions (9)

  • Definition 1.1
  • Definition 2.1
  • Theorem 2.1
  • Remark 2.2
  • Proposition C.1
  • Lemma C.2
  • proof
  • Lemma C.3
  • proof