Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning

Nirav Diwan, Tolga Ergen, Dongsub Shim, Honglak Lee

TL;DR

The paper addresses the challenge of obtaining high-quality preference data for Direct Preference Optimization (DPO) by showing that the reference-model probability space naturally signals clear preference signals. It introduces a reference-model guided sampling method that selects pairs with large probability gaps, reducing data requirements to 30-50% while maintaining or improving MT-Bench performance, and achieving especially strong gains on technical tasks like coding, math, and reasoning. Through experiments on Ultrafeedback with multiple LLN architectures and rigorous ablations, the authors demonstrate data-efficient improvements that generalize across models and hyperparameters. The work suggests a practical, annotation-light path to better preference learning and provides insights into how probability gaps in the reference space relate to alignment signal strength, with potential impact on scalable LLM alignment pipelines.

Abstract

Direct Preference Optimization (DPO) has emerged as a de-facto approach for aligning language models with human preferences. Recent work has shown DPO's effectiveness relies on training data quality. In particular, clear quality differences between preferred and rejected responses enhance learning performance. Current methods for identifying and obtaining such high-quality samples demand additional resources or external models. We discover that reference model probability space naturally detects high-quality training samples. Using this insight, we present a sampling strategy that achieves consistent improvements (+0.1 to +0.4) on MT-Bench while using less than half (30-50%) of the training data. We observe substantial improvements (+0.4 to +0.98) for technical tasks (coding, math, and reasoning) across multiple models and hyperparameter settings.

Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning

TL;DR

The paper addresses the challenge of obtaining high-quality preference data for Direct Preference Optimization (DPO) by showing that the reference-model probability space naturally signals clear preference signals. It introduces a reference-model guided sampling method that selects pairs with large probability gaps, reducing data requirements to 30-50% while maintaining or improving MT-Bench performance, and achieving especially strong gains on technical tasks like coding, math, and reasoning. Through experiments on Ultrafeedback with multiple LLN architectures and rigorous ablations, the authors demonstrate data-efficient improvements that generalize across models and hyperparameters. The work suggests a practical, annotation-light path to better preference learning and provides insights into how probability gaps in the reference space relate to alignment signal strength, with potential impact on scalable LLM alignment pipelines.

Abstract

Direct Preference Optimization (DPO) has emerged as a de-facto approach for aligning language models with human preferences. Recent work has shown DPO's effectiveness relies on training data quality. In particular, clear quality differences between preferred and rejected responses enhance learning performance. Current methods for identifying and obtaining such high-quality samples demand additional resources or external models. We discover that reference model probability space naturally detects high-quality training samples. Using this insight, we present a sampling strategy that achieves consistent improvements (+0.1 to +0.4) on MT-Bench while using less than half (30-50%) of the training data. We observe substantial improvements (+0.4 to +0.98) for technical tasks (coding, math, and reasoning) across multiple models and hyperparameter settings.
Paper Structure (23 sections, 3 equations, 2 figures, 1 table)

This paper contains 23 sections, 3 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Background
  4. Direct Preference Optimization
  5. Preference Clarity
  6. Reference-model based sampling
  7. Method.
  8. Reference Model as Quality Detector.
  9. Experiments
  10. Dataset.
  11. Models and Hyperparameters
  12. Evaluation
  13. Results and Discussion
  14. Performance
  15. Hyperparameters
  16. ...and 8 more sections

Figures (2)

  • Figure 1: Relationship between sampling threshold ($\delta$) and preference clarity for Ultrafeedback using the fine-tuned LLAMA-3 8B as the reference model. The solid blue line is the preference clarity between preferred and rejected responses calculated using the difference of Ultrafeedback's preference scores. The dashed orange line (log scale) shows the available training pairs at each threshold. Preference pairs at a higher sampling threshold show clearer preferences, indicating that the reference model can identify high-quality preference pairs even when it incorrectly attributes the correct response.
  • Figure 2: Performance improvements across different task categories for the best version of sampling approach , measured by increase in MT-bench scores. Technical tasks (Coding, Math & Reasoning) show substantially larger gains compared to general tasks.