Inverse Delayed Reinforcement Learning
Simon Sinong Zhan, Qingyuan Wu, Zhian Ruan, Frank Yang, Philip Wang, Yixuan Wang, Ruochen Jiao, Chao Huang, Qi Zhu
TL;DR
The paper tackles inverse reinforcement learning when expert trajectories are contaminated by observation delays. It proposes IDRL, an off-policy adversarial framework that augments the state with past actions to form $x_t$ and learns reward features via a discriminator to guide policy optimization. Theoretical analysis provides Lipschitz-based bounds showing augmented-state IRL outperforms delay-based observations, and empirical results on MuJoCo demonstrate robust recovery of expert behavior under varying delays and limited demonstrations. This approach offers practical impact for delay-prone cyber-physical systems by enabling more reliable imitation learning without requiring perfectly synchronized observations.
Abstract
Inverse Reinforcement Learning (IRL) has demonstrated effectiveness in a variety of imitation tasks. In this paper, we introduce an IRL framework designed to extract rewarding features from expert trajectories affected by delayed disturbances. Instead of relying on direct observations, our approach employs an efficient off-policy adversarial training framework to derive expert features and recover optimal policies from augmented delayed observations. Empirical evaluations in the MuJoCo environment under diverse delay settings validate the effectiveness of our method. Furthermore, we provide a theoretical analysis showing that recovering expert policies from augmented delayed observations outperforms using direct delayed observations.