Continuous Mean-Zero Disagreement-Regularized Imitation Learning (CMZ-DRIL)
Noah Ford, Ryan W. Gardner, Austin Juhl, Nathan Larson
TL;DR
CMZ-DRIL addresses data-efficiency in imitation learning by turning ensemble disagreement into a continuous, mean-zero reward, enabling RL without environment-specific rewards. It employs a two-stage training process: pretraining on expert data to form an uncertainty-aware ensemble, followed by PPO optimization using a reward $r_i = -\alpha (u_i - \bar{u}_i)$ where $\bar{u}_i$ is an exponential moving average. Compared to Behavioral Cloning and DRIL, CMZ-DRIL delivers notable improvements in reward and trajectory similarity across PyUXV and MuJoCo tasks, closing about half the gap to the true environmental reward. The approach highlights the value of staying within data-rich regions and leveraging uncertainty to guide learning, with potential as a bootstrap mechanism for developmentally-inspired AI when demonstrations are scarce.
Abstract
Machine-learning paradigms such as imitation learning and reinforcement learning can generate highly performant agents in a variety of complex environments. However, commonly used methods require large quantities of data and/or a known reward function. This paper presents a method called Continuous Mean-Zero Disagreement-Regularized Imitation Learning (CMZ-DRIL) that employs a novel reward structure to improve the performance of imitation-learning agents that have access to only a handful of expert demonstrations. CMZ-DRIL uses reinforcement learning to minimize uncertainty among an ensemble of agents trained to model the expert demonstrations. This method does not use any environment-specific rewards, but creates a continuous and mean-zero reward function from the action disagreement of the agent ensemble. As demonstrated in a waypoint-navigation environment and in two MuJoCo environments, CMZ-DRIL can generate performant agents that behave more similarly to the expert than primary previous approaches in several key metrics.