Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimal Design for Human Preference Elicitation

Subhojyoti Mukherjee, Anusha Lalitha, Kousha Kalantari, Aniket Deshmukh, Ge Liu, Yifei Ma, Branislav Kveton

TL;DR

The paper tackles efficient elicitation of human Preferences to learn ranking over $L$ lists of $K$ items under a budget $n$. It introduces a matrix-valued generalization of the Kiefer–Wolfowitz optimal design, enabling a non-adaptive policy $\pi_*$ that optimally gathers information for both absolute and ranking feedback. The Dope family of algorithms uses this design to collect data and learn linear preference models with strong statistical guarantees: $\tilde{O}(d^2/n)$ prediction error for absolute feedback and $\tilde{O}(K^6(d^2)/n)$ for ranking feedback, along with exponential decay in ranking loss with budget. Empirically, Dope outperforms baselines on synthetic data and real QA-style datasets (Nectar and Anthropic), demonstrating practical applicability for offline reward-model training and preference learning in AI systems.

Abstract

Learning of preference models from human feedback has been central to recent advances in artificial intelligence. Motivated by the cost of obtaining high-quality human annotations, we study efficient human preference elicitation for learning preference models. The key idea in our work is to generalize optimal designs, an approach to computing optimal information-gathering policies, to lists of items that represent potential questions with answers. The policy is a distribution over the lists and we elicit preferences from them proportionally to their probabilities. To show the generality of our ideas, we study both absolute and ranking feedback models on items in the list. We design efficient algorithms for both and analyze them. Finally, we demonstrate that our algorithms are practical by evaluating them on existing question-answering problems.

Optimal Design for Human Preference Elicitation

TL;DR

The paper tackles efficient elicitation of human Preferences to learn ranking over lists of items under a budget . It introduces a matrix-valued generalization of the Kiefer–Wolfowitz optimal design, enabling a non-adaptive policy that optimally gathers information for both absolute and ranking feedback. The Dope family of algorithms uses this design to collect data and learn linear preference models with strong statistical guarantees: prediction error for absolute feedback and for ranking feedback, along with exponential decay in ranking loss with budget. Empirically, Dope outperforms baselines on synthetic data and real QA-style datasets (Nectar and Anthropic), demonstrating practical applicability for offline reward-model training and preference learning in AI systems.

Abstract

Learning of preference models from human feedback has been central to recent advances in artificial intelligence. Motivated by the cost of obtaining high-quality human annotations, we study efficient human preference elicitation for learning preference models. The key idea in our work is to generalize optimal designs, an approach to computing optimal information-gathering policies, to lists of items that represent potential questions with answers. The policy is a distribution over the lists and we elicit preferences from them proportionally to their probabilities. To show the generality of our ideas, we study both absolute and ranking feedback models on items in the list. We design efficient algorithms for both and analyze them. Finally, we demonstrate that our algorithms are practical by evaluating them on existing question-answering problems.
Paper Structure (24 sections, 9 theorems, 71 equations, 2 figures, 3 tables, 2 algorithms)

This paper contains 24 sections, 9 theorems, 71 equations, 2 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Setting
  3. Optimal Design and Matrix Kiefer-Wolfowitz
  4. Learning with Absolute Feedback
  5. Algorithm Dope
  6. Maximum Prediction Error Under Absolute Feedback
  7. Ranking Loss Under Absolute Feedback
  8. Learning with Ranking Feedback
  9. Algorithm Dope
  10. Maximum Prediction Error Under Ranking Feedback
  11. Ranking Loss Under Ranking Feedback
  12. Experiments
  13. Conclusions
  14. Proofs
  15. Proof of \ref{['thm:matrix kiefer-wolfowitz']}
  16. ...and 9 more sections

Key Result

Theorem 1

Let $M \geq 1$ be an integer and $\mathbf{A}_1, \dots, \mathbf{A}_L \in \mathbb{R}^{d \times M}$ be $L$ matrices whose column space spans $\mathbb{R}^d$. Then the following claims are equivalent: Furthermore, there exists a minimizer $\pi_*$ of $g(\pi)$ such that $|\mathrm{supp} \left(\pi_*\right)| \leq d (d + 1) / 2$.

Figures (2)

  • Figure 1: Ranking loss of all compared methods as a function of the number of rounds. The error bars are one standard error of the estimates.
  • Figure : Dope for absolute feedback.

Theorems & Definitions (15)

  • Theorem 1: Matrix Kiefer-Wolfowitz
  • proof
  • Lemma 2
  • Theorem 3: Maximum prediction error
  • Theorem 4: Ranking loss
  • proof
  • Theorem 5: Maximum prediction error
  • Theorem 6: Ranking loss
  • proof
  • Lemma 7
  • ...and 5 more