Offline Imitation Learning by Controlling the Effective Planning Horizon

TL;DR

The paper tackles offline imitation learning with limited expert demonstrations and suboptimal offline data, identifying that naïvely reducing the discount factor $\gamma$ can worsen performance due to distribution mismatches in discriminator training. It derives a KL-based visitation-matching objective for offline IL, then shows how to compute an optimal weight $\zeta^*$ and a dual variable $\nu$ to obtain a practical objective; a discriminator trained on discounted versus undiscounted data motivates the need for IGI. To address this, the authors introduce Inverse Geometric Initial state sampling (IGI), which reweights initial states so that the expert and empirical visitation distributions align, enabling stable learning when $\gamma$ is varied. Empirically, IGI improves over existing offline IL methods like DemoDICE and SMODICE on both finite/discrete and continuous MuJoCo tasks across multiple discount factors and data compositions, validating the theoretical trade-offs and underscoring the method’s robustness and practical impact for offline IL.

Abstract

In offline imitation learning (IL), we generally assume only a handful of expert trajectories and a supplementary offline dataset from suboptimal behaviors to learn the expert policy. While it is now common to minimize the divergence between state-action visitation distributions so that the agent also considers the future consequences of an action, a sampling error in an offline dataset may lead to erroneous estimates of state-action visitations in the offline case. In this paper, we investigate the effect of controlling the effective planning horizon (i.e., reducing the discount factor) as opposed to imposing an explicit regularizer, as previously studied. Unfortunately, it turns out that the existing algorithms suffer from magnified approximation errors when the effective planning horizon is shortened, which results in a significant degradation in performance. We analyze the main cause of the problem and provide the right remedies to correct the algorithm. We show that the corrected algorithm improves on popular imitation learning benchmarks by controlling the effective planning horizon rather than an explicit regularization.

Figures (13)

• Figure 1: Learning curves of DemoDICE kim2021demodice and SMODICE ma2022versatile over various choices of discount factor $\gamma$. We use $\mathcal{D}^O$ consisting of 400 expert trajectories and 800 random-policy trajectories, and $\mathcal{D}^E$ consisting of 1 expert trajectory. Evaluations are averaged over 3 seeds in a Hopper-v2 environment, and they are normalized so that 0 corresponds to the average score of the random-policy dataset, and 100 corresponds to the average score of the expert policy dataset.
• Figure 2: Toy infinite horizon MDP example with 3 states and 2 actions. All transitions are deterministic and shown with the arrows. $s_0$ is initial state, and $g$ is absorbing state. We indicate the probability of taking an action based on the corresponding policy on the arrows in the figure. (a) represents expert policy and (b) is for learned policy with parameter $\theta$.
• Figure 3: Learning curves of our method in HalfCheetah-v2. Here, suboptimal dataset is consisted of 100 expert trajectories and 800 random-policy trajectories. We applied moving average with 3 seeds. The plot on the left is the result of sampling $\tilde{p}_0(s\vert t)$ from expert dataset, and the right is the result of sampling from total dataset.
• Figure 4: Performance of our algorithm (IGI) and baselines according to different $\gamma$s on HalfCheetah-v2 with D2 dataset. Here, the shaded area shows the standard error of the normalized evaluation over 3 seeds.
• Figure 5: The trend of evaluation result with various discount factors in different settings. We plot the normalized average evaluation over 1000 random seeds.

Theorems & Definitions (9)

• theorem thmcountertheorem
• lemma thmcounterlemma
• proof
• lemma thmcounterlemma
• proof
• lemma thmcounterlemma
• proof
• theorem thmcountertheorem
• proof