Dual Active Learning for Reinforcement Learning from Human Feedback

TL;DR

This work tackles RLHF reward learning under costly, heterogeneous human feedback by formulating a dual active learning approach that simultaneously selects conversations and teachers using $D$-optimal design. It introduces a context-dependent heterogeneous teacher model and couples active learning with pessimistic offline RL to produce robust policies, with theoretical guarantees of asymptotic $D$-optimality and a $O(1/\sqrt{T})$ sub-optimality rate. Empirically, the method yields tighter reward estimations and improved policy performance on both simulations and large-language-model benchmarks (Anthropic, UltraFeedback), while demonstrating data-efficiency gains via batch querying. The framework significantly reduces labeling costs and enhances alignment of LLMs to human preferences, with practical impact for scalable, robust RLHF pipelines.

Abstract

Aligning large language models (LLMs) with human preferences is critical to recent advances in generative artificial intelligence. Reinforcement learning from human feedback (RLHF) is widely applied to achieve this objective. A key step in RLHF is to learn the reward function from human feedback. However, human feedback is costly and time-consuming, making it essential to collect high-quality conversation data for human teachers to label. Additionally, different human teachers have different levels of expertise. It is thus critical to query the most appropriate teacher for their opinions. In this paper, we use offline reinforcement learning (RL) to formulate the alignment problem. Motivated by the idea of $D$-optimal design, we first propose a dual active reward learning algorithm for the simultaneous selection of conversations and teachers. Next, we apply pessimistic RL to solve the alignment problem, based on the learned reward estimator. Theoretically, we show that the reward estimator obtained through our proposed adaptive selection strategy achieves minimal generalized variance asymptotically, and prove that the sub-optimality of our pessimistic policy scales as $O(1/\sqrt{T})$ with a given sample budget $T$. Through simulations and experiments on LLMs, we demonstrate the effectiveness of our algorithm and its superiority over state-of-the-arts.

Figures (8)

• Figure 1: Schematic representation of the conversation and teacher selection process. The goal is to select $T$ conversations from the conversation set and query a teacher $\beta_t$ from the teacher set for their preference $y_t$ between two responses for each selected conversation. The reward estimator $\widehat{\theta}_T$ is obtained based on the collected information $\{(z_t, \beta_t, y_t)\}_{t=1}^T$ where $z$ is a shorthand for $\phi(x,a^{(1)})-\phi(x,a^{(0)})$.
• Figure 2: Confidence ellipsoid (gray area) around the estimated parameter vector $\widehat{\theta}$ in two dimensions. The lengths of the principal axes (dashed lines) are negatively related to the eigenvalues $\lambda_1, \lambda_2$ of $M(\xi_T, \theta_*)$. Maximizing $\lambda_1\lambda_2$ (equal to maximizing $\det M(\xi_T, \theta_*)$ minimizes the ellipsoid and thus constrains $\widehat{\theta}$ to be close to $\theta_*$.
• Figure 3: Estimation error of MLE and sub-optimality gaps of pessimistic and greedy policies.
• Figure 4: Sub-optimality gaps for all policies. The three subplots show the sub-optimality gap when the batch size $K$ is 1, 50 and 100, respectively.
• Figure 5: Sub-optimality gap for Our Proposal at different ranges of teacher rationality.

Theorems & Definitions (3)

• proof
• proof
• proof