SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning

TL;DR

SURF tackles the data-inefficiency of preference-based RL by leveraging unlabeled experiences through pseudo-labeling and a novel temporal cropping data augmentation. By integrating semi-supervised reward learning with a consistency-inducing augmentation, SURF plug-ins into existing preference-based methods to dramatically reduce the number of human preferences required. Empirical results on Meta-world and DMControl show substantial improvements in feedback efficiency, often matching or nearing performance with many fewer queries, including in visual-pcontrol settings. This approach broadens the practical applicability of reward-learning in RL where reward engineering is challenging or costly.

Abstract

Preference-based reinforcement learning (RL) has shown potential for teaching agents to perform the target tasks without a costly, pre-defined reward function by learning the reward with a supervisor's preference between the two agent behaviors. However, preference-based learning often requires a large amount of human feedback, making it difficult to apply this approach to various applications. This data-efficiency problem, on the other hand, has been typically addressed by using unlabeled samples or data augmentation techniques in the context of supervised learning. Motivated by the recent success of these approaches, we present SURF, a semi-supervised reward learning framework that utilizes a large amount of unlabeled samples with data augmentation. In order to leverage unlabeled samples for reward learning, we infer pseudo-labels of the unlabeled samples based on the confidence of the preference predictor. To further improve the label-efficiency of reward learning, we introduce a new data augmentation that temporally crops consecutive subsequences from the original behaviors. Our experiments demonstrate that our approach significantly improves the feedback-efficiency of the state-of-the-art preference-based method on a variety of locomotion and robotic manipulation tasks.

Figures (9)

• Figure 1: Overview of SURF. (a) We leverage unlabeled experiences by generating pseudo-labels $\widehat{y}$ from the preference predictor $P_\psi$ in (\ref{['eq:pref_model']}). To mitigate the negative effects from this semi-supervised learning, we only utilize pseudo-labels when the confidence of the predictor is higher than threshold $\tau$. (b) Given two segments $(\sigma^0,\sigma^1)$, we generate augmented segments $({\widehat{\sigma}}^0, {\widehat{\sigma}}^1)$ by cropping the subsequence from each segment.
• Figure 2: Learning curves on robotic manipulation tasks as measured on the success rate. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs.
• Figure 3: Learning curves on locomotion tasks as measured on the ground truth reward. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs.
• Figure 4: Ablation study on Walker-walk. (a) Contribution of each technique in SURF, i.e., semi-supervised learning (SSL) and temporal cropping (TC). (b) Effects of query size. (c) Comparison of augmentation methods. The results show the mean and standard deviation averaged over five runs.
• Figure 5: Hyperparameter analysis on Walker-walk using 100 preference queries. The results show the mean and standard deviation averaged over five runs.