Comparing Few to Rank Many: Active Human Preference Learning using Randomized Frank-Wolfe

TL;DR

This work tackles learning to rank $N$ items from limited $K$-way human feedback under a Plackett-Luce model by casting data collection as a D-optimal design problem. To scale to large $N$ and $K$, it introduces DopeWolfe, a randomized Frank-Wolfe algorithm with low-rank and sparse updates and caching, enabling efficient selection of a small set of informative queries. Theoretical results provide generalization and convergence guarantees for the randomized FW framework on LHSCB problems, and empirical experiments on synthetic and real-world NLP datasets show improved ranking performance and substantial runtime improvements over baselines. The approach is applicable to RLHF-style reward learning and broader ranking tasks where only a small fraction of items can be evaluated, enabling faster, more informative human feedback collection.

Abstract

We study learning of human preferences from a limited comparison feedback. This task is ubiquitous in machine learning. Its applications such as reinforcement learning from human feedback, have been transformational. We formulate this problem as learning a Plackett-Luce model over a universe of $N$ choices from $K$-way comparison feedback, where typically $K \ll N$. Our solution is the D-optimal design for the Plackett-Luce objective. The design defines a data logging policy that elicits comparison feedback for a small collection of optimally chosen points from all ${N \choose K}$ feasible subsets. The main algorithmic challenge in this work is that even fast methods for solving D-optimal designs would have $O({N \choose K})$ time complexity. To address this issue, we propose a randomized Frank-Wolfe (FW) algorithm that solves the linear maximization sub-problems in the FW method on randomly chosen variables. We analyze the algorithm, and evaluate it empirically on synthetic and open-source NLP datasets.

Figures (5)

• Figure 1: Mean KDtau metric on datasets with synthetic feedback. $k$-medoids fails for the NECTAR dataset due to excessive memory requirements (2 TB and 8 TB of RAM, respectively).
• Figure 2: Ranking models learned through samples selected by $\tt DopeWolfe$ achieve lesser ranking loss than the ones learned through samples selected uniformly at random.
• Figure 3: Mean NDCG metric (higher is better) on BEIR-COVID, TREC-DL and NECTAR datasets with synthetic feedback.
• Figure 4: BEIR-COVID and TREC-DL datasets with synthetic feedback: Mean KDtau and NDCG metric on Datasets for $K=4$. On average, $\tt DopeWolfe$ achieves better performance than Uniform sampling and DBSCAN.
• Figure 5: BEIR-COVID and TREC-DL datasets with real feedback: Models learned through ranking feedback collected on $\tt DopeWolfe$ samples achieve higher NDCG@10 than the ones learned on Uniformly selected samples.

Theorems & Definitions (12)

• Proposition 1
• proof
• Proposition 2
• proof
• Theorem 3
• Corollary 4
• Definition 5
• Proposition 6: zhao2023analysisNesterov
• Theorem 7
• proof