Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Understanding Valuable Preference Data for Large Language Model Alignment

Zizhuo Zhang, Qizhou Wang, Shanshan Ye, Jianing Zhu, Jiangchao Yao, Bo Han, Masashi Sugiyama

TL;DR

The paper argues that the value of human preference data for large language model alignment is not intrinsic to the data but depends on the specific model and training setup. It introduces Truncated Influence Function (TIF) to identify mid-range, high-signal data while avoiding validation overfitting, and develops two efficient proxies LossDiff and IRM, combined into LossDiff-IRM for data selection. The findings show medium-IF data are most beneficial and LossDiff-IRM matches or exceeds full-data performance with only 50–65% data across multiple LLMs and alignment methods, with robustness to validation noise. The work demonstrates a scalable, model-aware data curation approach that challenges model-agnostic data filtering and supports a less-is-more strategy for RLHF and direct preference optimization.

Abstract

Large language model (LLM) alignment is typically achieved through learning from human preference comparisons, making the quality of preference data critical to its success. Existing studies often pre-process raw training datasets to identify valuable preference pairs using external reward models or off-the-shelf LLMs, achieving improved overall performance but rarely examining whether individual, selected data point is genuinely beneficial. We assess data quality through individual influence on validation data using our newly proposed truncated influence function (TIF), which mitigates the over-scoring present in traditional measures and reveals that preference data quality is inherently a property of the model. In other words, a data pair that benefits one model may harm another. This leaves the need to improve the preference data selection approaches to be adapting to specific models. To this end, we introduce two candidate scoring functions (SFs) that are computationally simpler than TIF and positively correlated with it. They are also model dependent and can serve as potential indicators of individual data quality for preference data selection. Furthermore, we observe that these SFs inherently exhibit errors when compared to TIF. To this end, we combine them to offset their diverse error sources, resulting in a simple yet effective data selection rule that enables the models to achieve a more precise selection of valuable preference data. We conduct experiments across diverse alignment benchmarks and various LLM families, with results demonstrating that better alignment performance can be achieved using less data, showing the generality of our findings and new methods.

Towards Understanding Valuable Preference Data for Large Language Model Alignment

TL;DR

The paper argues that the value of human preference data for large language model alignment is not intrinsic to the data but depends on the specific model and training setup. It introduces Truncated Influence Function (TIF) to identify mid-range, high-signal data while avoiding validation overfitting, and develops two efficient proxies LossDiff and IRM, combined into LossDiff-IRM for data selection. The findings show medium-IF data are most beneficial and LossDiff-IRM matches or exceeds full-data performance with only 50–65% data across multiple LLMs and alignment methods, with robustness to validation noise. The work demonstrates a scalable, model-aware data curation approach that challenges model-agnostic data filtering and supports a less-is-more strategy for RLHF and direct preference optimization.

Abstract

Large language model (LLM) alignment is typically achieved through learning from human preference comparisons, making the quality of preference data critical to its success. Existing studies often pre-process raw training datasets to identify valuable preference pairs using external reward models or off-the-shelf LLMs, achieving improved overall performance but rarely examining whether individual, selected data point is genuinely beneficial. We assess data quality through individual influence on validation data using our newly proposed truncated influence function (TIF), which mitigates the over-scoring present in traditional measures and reveals that preference data quality is inherently a property of the model. In other words, a data pair that benefits one model may harm another. This leaves the need to improve the preference data selection approaches to be adapting to specific models. To this end, we introduce two candidate scoring functions (SFs) that are computationally simpler than TIF and positively correlated with it. They are also model dependent and can serve as potential indicators of individual data quality for preference data selection. Furthermore, we observe that these SFs inherently exhibit errors when compared to TIF. To this end, we combine them to offset their diverse error sources, resulting in a simple yet effective data selection rule that enables the models to achieve a more precise selection of valuable preference data. We conduct experiments across diverse alignment benchmarks and various LLM families, with results demonstrating that better alignment performance can be achieved using less data, showing the generality of our findings and new methods.
Paper Structure (33 sections, 1 theorem, 18 equations, 12 figures, 13 tables)

This paper contains 33 sections, 1 theorem, 18 equations, 12 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Analysis: Truncated Influence Function (TIF)
  4. Influence Function
  5. Analysis: Truncated Influence Function (TIF)
  6. Methodology: LossDiff-IRM Data Selection
  7. Approximation Proxy of TIF
  8. LossDiff-IRM Preference Data Selection
  9. Experiments
  10. Experimental Setup
  11. Experimental Results
  12. Main Performance of Loss-IRM for Preference Data Selection
  13. Further Analysis
  14. Conclusion
  15. Background and Related Work
  16. ...and 18 more sections

Key Result

Lemma B.1

Assume that the validation-aligned model $\pi_{\theta_{\text{val}}}$ is obtained by performing a single gradient descent step from the current model $\pi_\theta$ on the loss, i.e., $\theta_{\text{val}} = \theta - \eta \nabla_\theta \mathcal{L}(\theta; \mathcal{D}_{\text{val}})$, where $\eta$ is the where $I(d; \pi_\theta; \mathcal{D}_{\text{val}})$ is the influence function defined in Eq. (eq:IF)

Figures (12)

  • Figure 1: Analysis of IF-based data partition on the Qwen3-0.6B-Base and Llama-3.2-1B.From Left to Right: Training DPO loss, evaluation DPO loss and evaluation reward margin. Subsets of {small, medium, large} are denoted by {blue, orange, green}, respectively, while different line styles indicate IF values computed using different epoch checkpoints. More analysis refers to Appendix \ref{['App:More_analysis_IF']}.
  • Figure 2: Correlation analysis on Qwen3-0.6B-Base and Llama-3.2-1B.Top: correlation between loss difference (LossDiff) and IF. Bottom: correlation between implicit reward margin (IRM) and IF.
  • Figure 3: Ablation study on WinRate. Comparison of LossDiff-IRM vs. its ablations that select data using only LossDiff ("w/ LossDiff") or only IRM ("w/ IRM"). Top: DPO training. Bottom: SLiC training. Single-Score results are provided in Appendix \ref{['App:ablation_single_score']}.
  • Figure 4: Exact TIF vs. LossDiff-IRM. Performance comparison of Pythia-410M trained on data selected by Exact TIF or LossDiff-IRM.
  • Figure 5: Impact of noisy validation set. Performance curves across different noise rates $r=\{0.0, 0.1, 0.2, 0.3, 0.4\}$; dashed lines denote performance of Full-Data training.
  • ...and 7 more figures

Theorems & Definitions (1)

  • Lemma B.1: Loss Difference (LossDiff) Correlates with Influence Function (IF)