Accelerating Inverse Reinforcement Learning with Expert Bootstrapping

TL;DR

This work tackles the inefficiency of inner-loop reinforcement learning in inverse reinforcement learning by introducing two expert-guided strategies. Expert Replay Bootstrapping (ERB) injects expert transitions into the learner's replay buffer to nudge policy learning toward high-reward regions, while Expert Q Bootstrapping (EQB) uses the expert's next action to refine Q-targets via a meta-policy that blends expert and learner actions. The methods are compatible with off-policy learners like SAC and yield substantial improvements on MaxEntIRL and f-IRL across MuJoCo tasks, dramatically reducing the steps required to reach a large fraction of deterministic expert performance (e.g., Hopper-v2). A toy tree-MDP analysis clarifies when and why these techniques help, particularly under hard-exploration conditions. Overall, ERB and EQB provide lightweight, broadly applicable accelerations for online IRL with clear practical benefits for sample-efficient imitation learning.

Abstract

Existing inverse reinforcement learning methods (e.g. MaxEntIRL, $f$-IRL) search over candidate reward functions and solve a reinforcement learning problem in the inner loop. This creates a rather strange inversion where a harder problem, reinforcement learning, is in the inner loop of a presumably easier problem, imitation learning. In this work, we show that better utilization of expert demonstrations can reduce the need for hard exploration in the inner RL loop, hence accelerating learning. Specifically, we propose two simple recipes: (1) placing expert transitions into the replay buffer of the inner RL algorithm (e.g. Soft-Actor Critic) which directly informs the learner about high reward states instead of forcing the learner to discover them through extensive exploration, and (2) using expert actions in Q value bootstrapping in order to improve the target Q value estimates and more accurately describe high value expert states. Our methods show significant gains over a MaxEntIRL baseline on the benchmark MuJoCo suite of tasks, speeding up recovery to 70\% of deterministic expert performance by 2.13x on HalfCheetah-v2, 2.6x on Ant-v2, 18x on Hopper-v2, and 3.36x on Walker2d-v2.

Figures (16)

• Figure 1: \ref{['fig:tree_mdp_diagram']}: An example toy tree MDP. Reward is only given for taking the leftmost actions. \ref{['fig:shaky_hands']}: Results for Behavior Cloning, MaxEntIRL, ERB, and EQB on the "shaky hands" MDP. Results are averaged over 5 seeds, and shaded areas represent standard deviation. The x-axis represents iterations, while the y-axis represents return.
• Figure 2: Top: Results for toy tree MDP task with branching factor 10 and $\eta_{\pi}$ set to different values. Results averaged over 5 seeds, shaded areas represent standard deviation. Bottom: Results for toy tree MDP task with branching factor 15. X-axis represents iterations, y-axis represents return.
• Figure 3: Deterministic returns on 4 MuJoCo tasks with a MaxEntIRL baseline. X-axis is iterations, where each iteration is 5000 policy learning environment steps.
• Figure 4: Deterministic returns on 4 MuJoCo tasks with a Jensen-Shannon Divergence $f$-IRL baseline. Each unit on the x-axis represents one iteration, or 5000 policy update environment steps.
• Figure 5: Stochastic returns on 4 MuJoCo tasks with a MaxEntIRL baseline. X-axis is iterations, where each iteration is 5000 policy learning environment steps.