Learning from Active Human Involvement through Proxy Value Propagation

TL;DR

This work addresses aligning RL agents with human preferences in a safety-conscious, data-efficient manner by enabling active human involvement without reward engineering. It introduces Proxy Value Propagation (PVP), a reward-free method that labels human actions with $+1$ and novice actions with $-1$, and propagates these proxy values through TD-learning to unlabeled exploration data, enabling human-like policy learning. PVP integrates with existing value-based RL (e.g., TD3, DQN) using a PV loss $J^{PV}$ and a TD loss $J^{TD}$, with a balanced usage of human and novice buffers, and achieves superior performance across continuous and discrete control tasks (MetaDrive, CARLA, GTA V, MiniGrid) and various control devices. The approach demonstrates improved learning efficiency, safer exploration, and favorable user experience, suggesting strong potential for broader deployment in human-in-the-loop autonomous systems.

Abstract

Learning from active human involvement enables the human subject to actively intervene and demonstrate to the AI agent during training. The interaction and corrective feedback from human brings safety and AI alignment to the learning process. In this work, we propose a new reward-free active human involvement method called Proxy Value Propagation for policy optimization. Our key insight is that a proxy value function can be designed to express human intents, wherein state-action pairs in the human demonstration are labeled with high values, while those agents' actions that are intervened receive low values. Through the TD-learning framework, labeled values of demonstrated state-action pairs are further propagated to other unlabeled data generated from agents' exploration. The proxy value function thus induces a policy that faithfully emulates human behaviors. Human-in-the-loop experiments show the generality and efficiency of our method. With minimal modification to existing reinforcement learning algorithms, our method can learn to solve continuous and discrete control tasks with various human control devices, including the challenging task of driving in Grand Theft Auto V. Demo video and code are available at: https://metadriverse.github.io/pvp

Figures (11)

• Figure 1: Illustration of Proxy Value Propagation. (A) Human oversees the agent's trial-and-error exploration with the environment. When the human subject does not intervene, the transitions will be recorded into the novice buffer $\mathcal{B}_n$. (B) When the human intervenes, both novice action $a_n$ and human action $a_h$ will be recorded into the human buffer $\mathcal{B}_h$ but only the human action will be applied to the environment. (C) In training, we use the human buffer to compute proxy value loss and propagate the human intent knowledge to all transitions via TD loss without access to the reward.
• Figure 2: Evaluation of PVP under four different environments with human control devices. For each environment, we plot the test-time performance curve of the agent trained by the proposed PVP and the RL counterpart TD3. The x-coordinate is the total number of environment interactions, which indicates the time steps the training agent (in RL method) or the human-agent system (in our method) experiences during training. Compared to the RL counterpart, the proposed method achieves much higher performance with superior learning efficiency.
• Figure 3: We visualize the action sequences generated by HACO and PVP agents in the same MetaDrive map who are trained to 40K steps. PVP has much smoother actions.
• Figure 4: Evolution of proxy values.
• Figure 5: MetaDrive Safety benchmark.

Theorems & Definitions (2)

• Theorem D.3: Upper bound of intent violation
• proof