DAPPER: Discriminability-Aware Policy-to-Policy Preference-Based Reinforcement Learning for Query-Efficient Robot Skill Acquisition

TL;DR

DAPPER tackles the low query efficiency of preference-based reinforcement learning in robot skill acquisition by introducing discriminability-aware, policy-to-policy queries. By training multiple policies from scratch and learning a discriminator to estimate human labeling discriminability, DAPPER prioritizes more easily distinguishable queries and fosters trajectory diversity, yielding faster convergence with fewer queries. The framework jointly optimizes a human-preference reward and a discriminability reward, enabling policies that are both highly rewarding and easily separable by humans. Experiments in simulation and real-world legged robots show improved query efficiency, robustness to discriminability levels, and promising sim-to-real transfer, with additional insights into the potential and limitations of VLM-based labeling and feature-design dependencies.

Abstract

Preference-based Reinforcement Learning (PbRL) enables policy learning through simple queries comparing trajectories from a single policy. While human responses to these queries make it possible to learn policies aligned with human preferences, PbRL suffers from low query efficiency, as policy bias limits trajectory diversity and reduces the number of discriminable queries available for learning preferences. This paper identifies preference discriminability, which quantifies how easily a human can judge which trajectory is closer to their ideal behavior, as a key metric for improving query efficiency. To address this, we move beyond comparisons within a single policy and instead generate queries by comparing trajectories from multiple policies, as training them from scratch promotes diversity without policy bias. We propose Discriminability-Aware Policy-to-Policy Preference-Based Efficient Reinforcement Learning (DAPPER), which integrates preference discriminability with trajectory diversification achieved by multiple policies. DAPPER trains new policies from scratch after each reward update and employs a discriminator that learns to estimate preference discriminability, enabling the prioritized sampling of more discriminable queries. During training, it jointly maximizes the preference reward and preference discriminability score, encouraging the discovery of highly rewarding and easily distinguishable policies. Experiments in simulated and real-world legged robot environments demonstrate that DAPPER outperforms previous methods in query efficiency, particularly under challenging preference discriminability conditions.

Figures (10)

• Figure 1: Overview of the proposed learning framework: Multiple policies with diverse feature representations are sequentially learned. Queries are generated by sampling from these policies based on human preference discriminability, ensuring easier labeling. This approach enhances query efficiency by increasing the number of discriminable queries.
• Figure 2: Learning process of DAPPER: Our framework consists of three steps. (1) Learning current policy $\pi_i$ with reward $r$. (2) Query collection. This step makes queries from current policy $\pi_i$ and sampled policy from policy dataset $\Pi$ based on discriminability. Then, trajectories of each policy are generated $\tau_x$ and $\tau_i$, and queries are labeled $y$. (3) Reward Update. This step trains the preference reward model $R^H$ and discriminator $D$ from query dataset $\mathcal{D}$.
• Figure 3: Analysis of actual human query responses for determining the threshold of preference discriminability $d^{\text{disc}}$ in the simulated annotator model: A questionnaire was conducted using a console system, as shown in (b). At the top of the screen, a reference video was displayed, showing a policy that matched a predefined target behavior. Below the reference video, two candidate policy videos were shown for comparison. Participants were asked to choose the trajectory closer to the reference by selecting one of three options: "Left is better", "Right is better", or "Can't decide". The candidate policies were prepared by exhaustively combining the listed values of $\text{Body Height [cm]} \in \{29, 37, 45, 53, 61\}$ and $\text{Body Incline Angle [degree]} \in \{-8, -4, 0, 4, 8\}$. For each combination of these parameters, a policy was trained using DAPPER under an idealized setting where $d^{\text{disc}}=0$, by optimizing the behavior to match the specified Body Height and Body Incline Angle values until convergence. Participants were instructed to compare only two features defined by $f^*$: Body Height and Body Incline Angle. Gait patterns were excluded. They could prioritize the features freely, but were told to consider both. Policy pairs were sampled from (a), and 105 predefined queries were presented sequentially. The queries were selected to ensure diversity in preference discriminability $d^{\text{disc}}$. All participants evaluated the same set of queries, and the order of the queries was randomized for each participant. The results shown in (c) represent the mean and standard deviation across five participants.
• Figure 4: Evaluation of query efficiency: (a) Comparison to previous works. Top: Discrimination rate of queries. Middle: Converge query number for reaching the feature error threshold ($d^{\text{pref}} < 0.02$), empirically defined by the convergence values of each method. If not converged within $2 \times 10^3$ queries, the value is clipped at $2 \times 10^3$. The threshold is shown as a red dashed line in the bottom plot. Bottom: Minimum values of feature error $d^{\text{pref}}$ achieved by the learned policies. "Small", "Medium", and "Large" denote levels of preference discriminability $d^{\text{disc}}$. Each curve shows the mean and standard deviation over five experiments. (b) Learned policy features per query number. This experiments use $d^{\text{disc}}$ = "Medium". The target feature $f^*$ is located at the intersection of the red dashed lines. The number of plotted points corresponds to the queries collected until the feature error falls below the threshold ($d^{\text{pref}} < 0.02$), indicated by the red frame.
• Figure 5: Performance comparison across different preference discriminability thresholds: The convergence query number denotes the point at which the learned policy reaches the feature error threshold ($d^{\text{pref}} < 0.02$). If convergence is not achieved within $2 \times 10^3$ queries, the value is clipped at $2 \times 10^3$. Each curve shows the mean and standard deviation over five experiments.