Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Nearly Optimal Active Preference Learning and Its Application to LLM Alignment

Yao Zhao, Kwang-Sung Jun

TL;DR

The paper tackles data efficiency in RLHF by reframing reward-model learning as instance-aware active preference learning. It introduces a novel experimental-design objective that yields instance-dependent label complexity guarantees for a logistic/BTL-style preference model, and complements this with a simple, practical greedy uncertainty-sampling method. The two methods are shown to reduce the number of human preference queries needed while maintaining reward-model accuracy on real RLHF datasets, outperforming several baselines. Practically, this work enables more scalable LLM alignment by making high-quality preference data collection more efficient and targeted to problem structure, with clear theoretical guarantees and empirical evidence.

Abstract

Aligning large language models (LLMs) depends on high-quality datasets of human preference labels, which are costly to collect. Although active learning has been studied to improve sample efficiency relative to passive collection, many existing approaches adopt classical experimental design criteria such as G- or D-optimality. These objectives are not tailored to the structure of preference learning, leaving open the design of problem-specific algorithms. In this work, we identify a simple intuition specific to preference learning that calls into question the suitability of these existing design objectives. Motivated by this insight, we propose two active learning algorithms. The first provides the first instance-dependent label complexity guarantee for this setting, and the second is a simple, practical greedy method. We evaluate our algorithm on real-world preference datasets and observe improved sample efficiency compared to existing methods.

Nearly Optimal Active Preference Learning and Its Application to LLM Alignment

TL;DR

The paper tackles data efficiency in RLHF by reframing reward-model learning as instance-aware active preference learning. It introduces a novel experimental-design objective that yields instance-dependent label complexity guarantees for a logistic/BTL-style preference model, and complements this with a simple, practical greedy uncertainty-sampling method. The two methods are shown to reduce the number of human preference queries needed while maintaining reward-model accuracy on real RLHF datasets, outperforming several baselines. Practically, this work enables more scalable LLM alignment by making high-quality preference data collection more efficient and targeted to problem structure, with clear theoretical guarantees and empirical evidence.

Abstract

Aligning large language models (LLMs) depends on high-quality datasets of human preference labels, which are costly to collect. Although active learning has been studied to improve sample efficiency relative to passive collection, many existing approaches adopt classical experimental design criteria such as G- or D-optimality. These objectives are not tailored to the structure of preference learning, leaving open the design of problem-specific algorithms. In this work, we identify a simple intuition specific to preference learning that calls into question the suitability of these existing design objectives. Motivated by this insight, we propose two active learning algorithms. The first provides the first instance-dependent label complexity guarantee for this setting, and the second is a simple, practical greedy method. We evaluate our algorithm on real-world preference datasets and observe improved sample efficiency compared to existing methods.
Paper Structure (21 sections, 13 theorems, 72 equations, 4 figures, 3 algorithms)

This paper contains 21 sections, 13 theorems, 72 equations, 4 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Confidence Interval
  5. Intuition
  6. Active Preference Learning via Experimental Design
  7. Optimal Allocation
  8. Comparison to G-Optimal and XY-Optimal Designs.
  9. Algorithm Design with Unknown $\theta^*$
  10. Greedy Algorithm
  11. Experiments
  12. Conclusion
  13. Proofs
  14. Proof of Theorem \ref{['thm:exp_design_complexity_logi']}
  15. Proof of Theorem \ref{['thm:correctness']}
  16. ...and 6 more sections

Key Result

Theorem 1

(Theorem 1 of jun2021improved) Let $\delta \le e^{-1}$. Let $\hat{\theta}_t$ be the solution of eq:ols_theta where, for every $s \in [t]$, $y_s$ is conditionally independent from $\{z_i\}_{i=1}^t \setminus \{z_s\}$ given $z_s$ (i.e., the $z_s$'s are a fixed design). Fix $z \in \mathbb{R}^d$ with $\| Define the event If then where is the Fisher information matrix at $\theta$.

Figures (4)

  • Figure 1: Intuition for our proposed method. An arm's sign is considered confidently determined (green) if its confidence interval does not overlap with zero, and uncertain (red) otherwise. The numbers on the plots are the width of each confidence interval.
  • Figure 2: Illustration of remaining uncertainty (in olive). The remaining uncertainty is positive only when a confidence interval contains the decision boundary (zero), indicating that the arm's sign is not yet confidently determined.
  • Figure 3: Comparison of different active preference learning methods on the Anthropic helpful and harmless dataset.
  • Figure 4: Comparison of different active preference learning methods on three datasets.

Theorems & Definitions (26)

  • Definition 1
  • Remark
  • Theorem 1
  • Remark
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Lemma 4.1
  • Theorem 5
  • proof
  • ...and 16 more