Efficient Active Imitation Learning with Random Network Distillation

TL;DR

This paper tackles the challenge of learning realistic policies in tasks lacking clear rewards by optimizing expert involvement. It introduces RND-DAgger, an active imitation learning method that uses Random Network Distillation to detect out-of-distribution states and trigger expert interventions only when necessary, while enforcing a minimal demonstration time to stabilize demonstrations. Through experiments in robotics and three video-game-inspired environments, RND-DAgger demonstrates competitive final performance, faster data efficiency, and substantially reduced expert burden compared to prior active-learning approaches. The work provides practical benefits for training autonomous agents in complex, high-variability settings and suggests pathways for scaling to more complex observation spaces and human-in-the-loop workflows.

Abstract

Developing agents for complex and underspecified tasks, where no clear objective exists, remains challenging but offers many opportunities. This is especially true in video games, where simulated players (bots) need to play realistically, and there is no clear reward to evaluate them. While imitation learning has shown promise in such domains, these methods often fail when agents encounter out-of-distribution scenarios during deployment. Expanding the training dataset is a common solution, but it becomes impractical or costly when relying on human demonstrations. This article addresses active imitation learning, aiming to trigger expert intervention only when necessary, reducing the need for constant expert input along training. We introduce Random Network Distillation DAgger (RND-DAgger), a new active imitation learning method that limits expert querying by using a learned state-based out-of-distribution measure to trigger interventions. This approach avoids frequent expert-agent action comparisons, thus making the expert intervene only when it is useful. We evaluate RND-DAgger against traditional imitation learning and other active approaches in 3D video games (racing and third-person navigation) and in a robotic locomotion task and show that RND-DAgger surpasses previous methods by reducing expert queries. https://sites.google.com/view/rnd-dagger

Figures (14)

• Figure 1: RND-DAgger overview. (a) The learner’s policy controls the agent until (b) our Random Network Distillation-based out-of-distribution (OOD) measure is triggered. Then (c) the expert takes control until the OOD measure gets lower than the threshold for at least W steps. (d) Finally, the current policy takes control of the agent to continue the episode, and can trigger the expert again later.
• Figure 2: DAgger
• Figure 3: Ensemble-DAgger's Condition
• Figure 4: RND-DAgger
• Figure 5: Illustration of the three different environments used in our experiments. The objective is to learn a good driving, navigating, and walking policy while minimizing expert interventions.