Larger or Smaller Reward Margins to Select Preferences for Alignment?

TL;DR

This work addresses inconsistent data quality assessments in offline preference learning for LLM alignment by introducing the alignment potential metric $M_{AP}$, which measures the gap between the target explicit reward margin and the model's current implicit reward margin. By unifying explicit and implicit reward perspectives, $M_{AP}$ guides the selection and generation of high-quality preference data, improving alignment performance across base models and optimization objectives, and extending naturally to self-play data generation frameworks like eva. The authors provide theoretical convergence insights under adversarial sampling and demonstrate empirical gains in both offline and self-play settings, including data-size and training-iteration scaling. The work offers a practical, robust metric for data quality that enhances RLHF workflows and reduces annotation noise, with implications for more efficient and scalable alignment of LLMs to human values.

Abstract

Preference learning is critical for aligning large language models (LLMs) with human values, with the quality of preference datasets playing a crucial role in this process. While existing metrics primarily assess data quality based on either explicit or implicit reward margins, they often provide contradictory evaluations for the same data. To address this issue, we introduce the alignment potential metric, which quantifies the gap from the model's current implicit reward margin to the target explicit reward margin, thereby estimating the model's potential to align with the preference data. Empirical results demonstrate that training on data selected by this metric consistently enhances alignment performance, surpassing existing metrics across different base models and optimization objectives. Furthermore, our method extends to self-play data generation frameworks, where the metric is used to identify high-quality data within the self-generated content by LLMs. Under this data generation scenario, our method surpasses current state-of-the-art (SOTA) results across various training settings and demonstrates continuous improvements in alignment performance as dataset size and training iterations increase.

Figures (11)

• Figure 1: (\ref{['fig:teaser_a']}) Contradiction of existing metrics. The left example, with large explicit and implicit reward margins, is rated as "high quality" by explicit reward margin but "low quality" by implicit reward margin. The right example, where both margins are small, is rated as "low quality" by the explicit margin but "high quality" by the implicit margin. In both cases, the implicit margins are already aligned with the explicit ones, making both data rated as "low quality" by our metric, i.e., no need for further training. (\ref{['fig:teaser_b']}) Enhanced performance by data selection. Llama-3-8b-instruct and Gemma-2-9b-it's performance on Top-40% data subset selected by different data metrics (uniform refers to uniformly sampling 40% data from the dataset), with our proposed metric achieving the highest results.
• Figure 2: Reward noise example from SimPO's dataset, where the less preferred response $y_l$ by RM is the correct answer. $M_+$ will mislabel this data as high-quality, which the $M_{AP}$ metric avoids.
• Figure 3: GPT-4 agreement with reward model's annotation on data selected by $M_+$ or $M_{AP}$ metric from https://huggingface.co/datasets/princeton-nlp/gemma2-ultrafeedback-armorm. The data selected by $M_{AP}$ has a notably higher agree rate than $M_+$, indicating less reward annotation noise.
• Figure 4: Performance of Llama-3-8b-instruct model trained on preference pairs selected by different metrics.
• Figure 5: Performance of Gemma-2-9b-it model trained on preference pairs selected by different metrics.

Theorems & Definitions (2)

• Theorem 3.1
• proof : Proof of Theorem \ref{['thm:convergence']}