Imitation Learning by State-Only Distribution Matching

TL;DR

A non-adversarial learning-from-observations approach, together with an interpretable convergence and performance metric, that minimizes the Kulback-Leibler divergence between the policy and expert state transition trajectories which can be optimized in a non- adversarial fashion.

Abstract

Imitation Learning from observation describes policy learning in a similar way to human learning. An agent's policy is trained by observing an expert performing a task. While many state-only imitation learning approaches are based on adversarial imitation learning, one main drawback is that adversarial training is often unstable and lacks a reliable convergence estimator. If the true environment reward is unknown and cannot be used to select the best-performing model, this can result in bad real-world policy performance. We propose a non-adversarial learning-from-observations approach, together with an interpretable convergence and performance metric. Our training objective minimizes the Kulback-Leibler divergence (KLD) between the policy and expert state transition trajectories which can be optimized in a non-adversarial fashion. Such methods demonstrate improved robustness when learned density models guide the optimization. We further improve the sample efficiency by rewriting the KLD minimization as the Soft Actor Critic objective based on a modified reward using additional density models that estimate the environment's forward and backward dynamics. Finally, we evaluate the effectiveness of our approach on well-known continuous control environments and show state-of-the-art performance while having a reliable performance estimator compared to several recent learning-from-observation methods.

Figures (8)

• Figure 1: Unkown true environment reward selection criteria: Relative reward for different amount of expert trajectories on continuous control environments. The best policies based on estimated convergence values were selected. The value 1 corresponds to expert policy performance.
• Figure 2: Best true environment reward selection criterion: Relative reward for different amount of expert trajectories on continuous control environments. The value 1 corresponds to expert policy performance.
• Figure 3: Best environment reward for ablation experiments. Relative reward for different amount of expert trajectories. The value 1 corresponds to expert policy performance.
• Figure 4: The policy loss, estimated reward and the environment test loss during training in the pybullet Ant environment using our proposed SOIL-TDM and the OPOLO, F-IRL, and FORM implementations with 4 expert trajectories.
• Figure 5: The policy loss, estimated reward and the environment test reward during training in the pybullet Hopper environment using our proposed SOIL-TDM and the OPOLO, F-IRL, and FORM implementations with 4 expert trajectories.