Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Active Learning for Direct Preference Optimization

Branislav Kveton, Xintong Li, Julian McAuley, Ryan Rossi, Jingbo Shang, Junda Wu, Tong Yu

TL;DR

This work introduces an active learning framework for Direct Preference Optimization (DPO) by linearizing the DPO objective at the final-layer representation and formulating a D-optimal design to select the most informative preferential feedback. It presents two practical algorithms, ADPO for online feedback and ADPO$^+$ for offline feedback, with theoretical guarantees showing the maximum logit error decays as ${\tilde{O}}(d/\sqrt{n})$ under a log-linear policy model. Empirically, the methods improve data efficiency and policy performance on both synthetic log-linear tasks and large language model settings, including real-world preference data. The results advance principled, information-theoretic data selection for aligning LLMs with human preferences, enabling more efficient RLHF and DPO pipelines.

Abstract

Direct preference optimization (DPO) is a form of reinforcement learning from human feedback (RLHF) where the policy is learned directly from preferential feedback. Although many models of human preferences exist, the critical task of selecting the most informative feedback for training them is under-explored. We propose an active learning framework for DPO, which can be applied to collect human feedback online or to choose the most informative subset of already collected feedback offline. We propose efficient algorithms for both settings. The key idea is to linearize the DPO objective at the last layer of the neural network representation of the optimized policy and then compute the D-optimal design to collect preferential feedback. We prove that the errors in our DPO logit estimates diminish with more feedback. We show the effectiveness of our algorithms empirically in the setting that matches our theory and also on large language models.

Active Learning for Direct Preference Optimization

TL;DR

This work introduces an active learning framework for Direct Preference Optimization (DPO) by linearizing the DPO objective at the final-layer representation and formulating a D-optimal design to select the most informative preferential feedback. It presents two practical algorithms, ADPO for online feedback and ADPO for offline feedback, with theoretical guarantees showing the maximum logit error decays as under a log-linear policy model. Empirically, the methods improve data efficiency and policy performance on both synthetic log-linear tasks and large language model settings, including real-world preference data. The results advance principled, information-theoretic data selection for aligning LLMs with human preferences, enabling more efficient RLHF and DPO pipelines.

Abstract

Direct preference optimization (DPO) is a form of reinforcement learning from human feedback (RLHF) where the policy is learned directly from preferential feedback. Although many models of human preferences exist, the critical task of selecting the most informative feedback for training them is under-explored. We propose an active learning framework for DPO, which can be applied to collect human feedback online or to choose the most informative subset of already collected feedback offline. We propose efficient algorithms for both settings. The key idea is to linearize the DPO objective at the last layer of the neural network representation of the optimized policy and then compute the D-optimal design to collect preferential feedback. We prove that the errors in our DPO logit estimates diminish with more feedback. We show the effectiveness of our algorithms empirically in the setting that matches our theory and also on large language models.

Paper Structure

This paper contains 25 sections, 6 theorems, 61 equations, 4 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Supervised Fine-Tuning
  4. Reinforcement Learning from Human Feedback
  5. Direct Preference Optimization
  6. Setting
  7. Algorithms
  8. Active DPO with Online Preferential Feedback
  9. Active DPO with Offline Preferential Feedback
  10. Analysis
  11. Main Result
  12. Discussion
  13. Experiments
  14. Log-Linear Policies
  15. LLM Policies
  16. ...and 10 more sections

Key Result

Lemma 1

Let $\pi(y \mid x; \theta)$ be a log-linear policy. Then the Hessian of $\mathcal{L}_\textsc{dpo}(\theta; \mathcal{S})$ in eq:negative dpo loglik with respect to $\theta$ is It is also positive semi-definite.

Figures (4)

  • Figure 1: Experiments with log-linear policies on the CIFAR-10 (first row) and CIFAR-100 (second row) datasets.
  • Figure 2: Experiments with LLM policies on the Nectar dataset. We use Llama-3.2 (first row) and Phi-3 (second row) models.
  • Figure 3: Experiments with log-linear policies on the CIFAR-10 dataset, with $\beta = 2$ (first row) and $\beta = 5$ (second row).
  • Figure 4: Experiments with log-linear policies on the CIFAR-10 (first row) and CIFAR-100 (second row) datasets with $\alpha = 0$ in \ref{['eq:ucb']}.

Theorems & Definitions (9)

  • Lemma 1
  • proof
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Lemma 5
  • proof
  • Lemma 6
  • proof