Positive-Unlabeled Reward Learning

TL;DR

This work tackles reward-learning challenges in RL by reframing both adversarial imitation and supervised reward learning as positive-unlabeled (PU) classification problems. By applying a large-scale PU risk estimator, the authors derive a unified Positive-Unlabeled Reward Learning (PURL) framework that addresses reward delusions and discriminator overfitting simultaneously. Empirical results across standard benchmarks and robotic tasks show that PURL improves robustness and performance over GAIL and supervised reward baselines, even under domain gaps. The approach suggests a versatile pathway for integrating PU learning with reinforcement learning and GAN-based imitation, with several promising directions for future work.

Abstract

Learning reward functions from data is a promising path towards achieving scalable Reinforcement Learning (RL) for robotics. However, a major challenge in training agents from learned reward models is that the agent can learn to exploit errors in the reward model to achieve high reward behaviors that do not correspond to the intended task. These reward delusions can lead to unintended and even dangerous behaviors. On the other hand, adversarial imitation learning frameworks tend to suffer the opposite problem, where the discriminator learns to trivially distinguish agent and expert behavior, resulting in reward models that produce low reward signal regardless of the input state. In this paper, we connect these two classes of reward learning methods to positive-unlabeled (PU) learning, and we show that by applying a large-scale PU learning algorithm to the reward learning problem, we can address both the reward under- and over-estimation problems simultaneously. Our approach drastically improves both GAIL and supervised reward learning, without any additional assumptions.

Figures (8)

• Figure 1: Left: Illustration of the reward delusion problem in supervised reward learning, along with our solution. The x-axis represents the state space of the task, and the y-axis represents reward. The black dots show annotated states used for training the reward model, and the red dashed line shows reward model predictions extended over the entire state space. There is a large region of the state space where the reward model vastly overestimates the true reward (shown in blue). Our solution in this case is to learn the function $D$, shown in green, that identifies regions of the state space where the reward model is a reliable estimate of the true reward. Right: Illustration of the overfitting problem of GAIL, along with our solution. Early in training ($t_0$) agents will generally fail to complete the intended task, and the discriminator can easily separate agent and expert experience. Later in training, when the agent is competent ($t_T$), the agent will often successfully complete the task, and the GAIL discriminator will overfit by focusing on irrelevant features. If we could detect success and failure for policy rollouts then we could train the discriminator to distinguish between them directly (middle column), but this information is generally not available. Our solution uses PU learning to train a success vs. failure classifier using positive (expert) and unlabeled (agent) data, without requiring success and failure annotations for policy rollouts.
• Figure 2: Results of the adversarial imitation learning methods on the three evaluation tasks. Each curve is the mean of 5 trials with confidence interval of 95%.
• Figure 3: Ablation study of adversarial imitation learning on the lifting task: Effect of choosing different positive class prior $\eta$ values on the (a) discriminator performance and (b) policy performance for nn-PUGAIL. (c) compares the discriminator behavior of GAIL and nn-PUGAIL over the course of the policy learning. Each curve is the mean of 5 trials with confidence interval of 95%.
• Figure 4: Results of supervised reward learning setting on the three evaluation tasks. Each curve is the mean of 5 trials with confidence interval of 95%.
• Figure 5: Effect of positive class prior $\eta$ values on the (a) reward prediction errors and (b) policy performance in nn-PURL. Results of PNRL and PRL are included for reference.