Imitation Learning via Off-Policy Distribution Matching

TL;DR

The paper introduces ValueDICE, an imitation learning algorithm that formulates distribution matching as an off-policy objective based on the Donsker-Varadhan representation of KL divergence. It casts the problem as a max-min optimization between a policy $\pi$ and a value-like function $\nu$, enabling direct off-policy learning from expert data without explicit reward signals or a separate RL step. The method achieves strong sample efficiency on both synthetic Ring MDPs and MuJoCo continuous-control benchmarks, outperforming or matching prior off-policy imitation methods, especially in data-scarce regimes. Practical contributions include replay-buffer regularization and initial-state augmentation to improve data efficiency and robustness in realistic imitation tasks.

Abstract

When performing imitation learning from expert demonstrations, distribution matching is a popular approach, in which one alternates between estimating distribution ratios and then using these ratios as rewards in a standard reinforcement learning (RL) algorithm. Traditionally, estimation of the distribution ratio requires on-policy data, which has caused previous work to either be exorbitantly data-inefficient or alter the original objective in a manner that can drastically change its optimum. In this work, we show how the original distribution ratio estimation objective may be transformed in a principled manner to yield a completely off-policy objective. In addition to the data-efficiency that this provides, we are able to show that this objective also renders the use of a separate RL optimization unnecessary.Rather, an imitation policy may be learned directly from this objective without the use of explicit rewards. We call the resulting algorithm ValueDICE and evaluate it on a suite of popular imitation learning benchmarks, finding that it can achieve state-of-the-art sample efficiency and performance.

Figures (4)

• Figure 1: Results of ValueDICE on a simple Ring MDP. Left: The expert data is sparse and only covers states 0, 1, and 2. Nevertheless, ValueDICE is able to learn a policy on all states to best match the observed expert state-action occupancies (the policy learns to always go to states 1 and 2). Right: The expert is stochastic. ValueDICE is able to learn a policy which successfully minimizes the true KL computed between $d^{\pi}$ and $d^{\mathrm{exp}}$.
• Figure 2: Comparison of algorithms given 1 expert trajectory. We use the original implementation of GAIL ho2016generative to produce GAIL and BC results.
• Figure 3: Comparison of algorithms given 10 expert trajectories. ValueDICE outperforms other methods. However, given this amount of data, BC can recover the expert policy as well.
• Figure 4: ValueDICE outperforms behavioral cloning given 1 trajectory even without replay regularization.