REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback

TL;DR

This work tackles reward misalignment in robotic reinforcement learning from human feedback by introducing agent-preference regularization (REBEL), a regularization term that captures the agent's own trajectory preferences to mitigate distribution shift between reward learning and policy execution. The authors formulate the RLHF problem as a bilevel optimization and derive a computationally tractable first-order reformulation, enabling efficient learning in large-scale robotic settings. Empirically, REBEL improves sample efficiency and performance over state-of-the-art preference-based baselines on DeepMind Control Suite locomotion tasks using simulated human teachers. The approach offers a principled path to safer, more aligned robotic policies by jointly accounting for human and agent preferences during reward learning, with limitations including potential sensitivity to hyperparameters and reliance on human-like preference signals.

Abstract

The effectiveness of reinforcement learning (RL) agents in continuous control robotics tasks is mainly dependent on the design of the underlying reward function, which is highly prone to reward hacking. A misalignment between the reward function and underlying human preferences (values, social norms) can lead to catastrophic outcomes in the real world especially in the context of robotics for critical decision making. Recent methods aim to mitigate misalignment by learning reward functions from human preferences and subsequently performing policy optimization. However, these methods inadvertently introduce a distribution shift during reward learning due to ignoring the dependence of agent-generated trajectories on the reward learning objective, ultimately resulting in sub-optimal alignment. Hence, in this work, we address this challenge by advocating for the adoption of regularized reward functions that more accurately mirror the intended behaviors of the agent. We propose a novel concept of reward regularization within the robotic RLHF (RL from Human Feedback) framework, which we refer to as \emph{agent preferences}. Our approach uniquely incorporates not just human feedback in the form of preferences but also considers the preferences of the RL agent itself during the reward function learning process. This dual consideration significantly mitigates the issue of distribution shift in RLHF with a computationally tractable algorithm. We provide a theoretical justification for the proposed algorithm by formulating the robotic RLHF problem as a bilevel optimization problem and developing a computationally tractable version of the same. We demonstrate the efficiency of our algorithm {\ours} in several continuous control benchmarks in DeepMind Control Suite \cite{tassa2018deepmind}.

Figures (2)

• Figure 1: This figure describes the preference-based robotic reinforcement learning framework from human feedback (RRLHF) lee2021pebble. We introduce a novel regularized reward learning into the existing RRLHF framework, which helps improve the state-of-the-art performance
• Figure 2: Learning curves on locomotion tasks as measured on the ground truth reward. The solid line and shaded regions, respectively, represent the mean and standard deviation across three runs.