Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment

Dongyoung Kim, Kimin Lee, Jinwoo Shin, Jaehyung Kim

TL;DR

<3-5 sentence high-level summary> SPA introduces Spread Preference Annotation, a method to align large language models with human preferences using only a small seed of labeled data. It combines direct preference judgments derived from model logits with iterative self-generated data expansion and a noise-aware self-refinement mechanism to spread human preference signals across iterations. The approach shows strong improvements on AlpacaEval 2.0 and MT-Bench with minimal ground-truth data, outperforming baselines that rely on external reward models or LLM-based judgments. SPA also demonstrates robustness across seeds and model families, indicating practical utility for data-scarce alignment tasks.

Abstract

Aligning large language models (LLMs) with human preferences becomes a key component to obtaining state-of-the-art performance, but it yields a huge cost to construct a large human-annotated preference dataset. To tackle this problem, we propose a new framework, Spread Preference Annotation with direct preference judgment (SPA), that boosts the alignment of LLMs using only a very small amount of human-annotated preference data. Our key idea is leveraging the human prior knowledge within the small (seed) data and progressively improving the alignment of LLM, by iteratively generating the responses and learning from them with the self-annotated preference data. To be specific, we propose to derive the preference label from the logits of LLM to explicitly extract the model's inherent preference. Compared to the previous approaches using external reward models or implicit in-context learning, we observe that the proposed approach is significantly more effective. In addition, we introduce a noise-aware preference learning algorithm to mitigate the risk of low quality within generated preference data. Our experimental results demonstrate that the proposed framework significantly boosts the alignment of LLMs. For example, we achieve superior alignment performance on AlpacaEval 2.0 with only 3.3% of the ground-truth preference labels in the Ultrafeedback data compared to the cases using the entire data or state-of-the-art baselines.

Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment

TL;DR

<3-5 sentence high-level summary> SPA introduces Spread Preference Annotation, a method to align large language models with human preferences using only a small seed of labeled data. It combines direct preference judgments derived from model logits with iterative self-generated data expansion and a noise-aware self-refinement mechanism to spread human preference signals across iterations. The approach shows strong improvements on AlpacaEval 2.0 and MT-Bench with minimal ground-truth data, outperforming baselines that rely on external reward models or LLM-based judgments. SPA also demonstrates robustness across seeds and model families, indicating practical utility for data-scarce alignment tasks.

Abstract

Aligning large language models (LLMs) with human preferences becomes a key component to obtaining state-of-the-art performance, but it yields a huge cost to construct a large human-annotated preference dataset. To tackle this problem, we propose a new framework, Spread Preference Annotation with direct preference judgment (SPA), that boosts the alignment of LLMs using only a very small amount of human-annotated preference data. Our key idea is leveraging the human prior knowledge within the small (seed) data and progressively improving the alignment of LLM, by iteratively generating the responses and learning from them with the self-annotated preference data. To be specific, we propose to derive the preference label from the logits of LLM to explicitly extract the model's inherent preference. Compared to the previous approaches using external reward models or implicit in-context learning, we observe that the proposed approach is significantly more effective. In addition, we introduce a noise-aware preference learning algorithm to mitigate the risk of low quality within generated preference data. Our experimental results demonstrate that the proposed framework significantly boosts the alignment of LLMs. For example, we achieve superior alignment performance on AlpacaEval 2.0 with only 3.3% of the ground-truth preference labels in the Ultrafeedback data compared to the cases using the entire data or state-of-the-art baselines.
Paper Structure (22 sections, 13 equations, 10 figures, 11 tables, 1 algorithm)

This paper contains 22 sections, 13 equations, 10 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. SPA: Spread Preference Annotation to boost Alignment of LLMs
  5. Direct preference judgment to align LLMs with self-generated data
  6. Self-refinement of generated preference data for effective learning
  7. Experiments
  8. Experimental setups
  9. Main results
  10. More analyses
  11. Conclusion
  12. Limitation and Societal Impact
  13. Limitation and future work
  14. Societal impact
  15. More Details of Experimental Setups
  16. ...and 7 more sections

Figures (10)

  • Figure 1: Illustration of the proposed SPA framework.SPA progressively improves the alignment of LLMs by iterating (1) the generation of new preference data and (2) the preference learning on the constructed data with self-refinement. Technical details are presented in Section \ref{['sec:method']}.
  • Figure 2: Summary of main result. Evaluation results on AlpacaEval 2.0 alpaca_eval. Our framework significantly improves the alignment of LLMs, without additional human preference data. See detailed results in Section \ref{['sec:experiments']}.
  • Figure 3: Improvements during iterations. Length control (LC.) win rate (%) measured by AlpacaEval 2.0 is consistently improved by SPA and it outperforms other baselines.
  • Figure 4: Improvements without seed data. Evaluation results on AlpacaEval 2.0 with Mistral-7B-instruct-v0.1 and SPA with no seed preference data.
  • Figure 5: Prompt used for judging preferences.
  • ...and 5 more figures