Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learning

TL;DR

This work tackles single-demonstration imitation learning by introducing TDIL, a method that learns a dense, dynamics-aware surrogate reward through a transition discriminator to guide an agent back toward expert proximity. By defining expert reachability and approximating the surrogate reward with a transition discriminator, TDIL provides richer credit signals than traditional IRL alone, and it can be trained concurrently with a SAC agent. The approach demonstrates expert-level performance on MuJoCo benchmarks and Adroit Door, enables blind model selection via relative rewards, and shows robustness through extensive ablations, including handling multiple demonstrations. The results highlight the practical impact of using environment dynamics to shape rewards when only a single expert trajectory is available, improving both learning efficiency and stability in real-world imitation tasks.

Abstract

In this paper, we focus on single-demonstration imitation learning (IL), a practical approach for real-world applications where acquiring multiple expert demonstrations is costly or infeasible and the ground truth reward function is not available. In contrast to typical IL settings with multiple demonstrations, single-demonstration IL involves an agent having access to only one expert trajectory. We highlight the issue of sparse reward signals in this setting and propose to mitigate this issue through our proposed Transition Discriminator-based IL (TDIL) method. TDIL is an IRL method designed to address reward sparsity by introducing a denser surrogate reward function that considers environmental dynamics. This surrogate reward function encourages the agent to navigate towards states that are proximal to expert states. In practice, TDIL trains a transition discriminator to differentiate between valid and non-valid transitions in a given environment to compute the surrogate rewards. The experiments demonstrate that TDIL outperforms existing IL approaches and achieves expert-level performance in the single-demonstration IL setting across five widely adopted MuJoCo benchmarks as well as the "Adroit Door" robotic environment.

Figures (13)

• Figure 1: A motivational grid-world example for comparing different IL methods trained with the single-demo IL setting. (a) depicts the expert's demonstration, denoted by blue arrows, while red lines represent impassable barriers, reflecting environmental dynamics. The green arrows symbolize the state-action pairs that are one step directed toward the expert states. Subfigures (b)-(d) present reward signals calculated through various methods: (b) using the basic IRL method (i.e., GAIL ho2016generative), (c) based on the L2 distance between the agent's and the expert's state-action pairs, and (d) through our proposed TDIL. Finally, subfigures (e)-(h) illustrate the actions calculated by averaging the directions represented by the logits for the discrete actions from the learned policy at distinct grid locations.
• Figure 2: The total steps per episode required for each agent to reach the goal in the grid-world, with a fixed limit of 50 steps. Lower values for "Steps per episode" indicate better efficiency.
• Figure 3: The approximation of $\Rsur$ through the use of the transition discriminator $D_\phi$.
• Figure 4: An overview of the TDIL method. Step 1: Agent-environment interaction. Step 2: Transition discriminator updates. Step 3: Generation of aggregated rewards. Step 4: Training an RL agent based on the generated reward signals.
• Figure 5: Normalized performance evaluation of different methodologies using the Oracle model selection under the single-demo setting.