Imitation Learning from Suboptimal Demonstrations via Meta-Learning An Action Ranker

TL;DR

Imitation learning often requires extensive expert demonstrations, which is costly. ILMAR addresses this by learning an action-ranker to weight suboptimal demonstrations through an advantage-based weighting scheme and a meta-goal that bi-linearly optimizes toward an expert-like policy. The method combines weighted behavior cloning with a discriminator-guided weighting, integrated in a bi-level optimization that enforces closeness to the expert via KL-based objectives. Empirical results on MuJoCo locomotion tasks show ILMAR achieving state-of-the-art performance among methods that utilize suboptimal demonstrations, with robust convergence when combined with vanilla loss and meta-goal. The work offers a practical framework for efficiently exploiting suboptimal data in imitation learning and provides theoretical convergence guarantees under mild assumptions.

Abstract

A major bottleneck in imitation learning is the requirement of a large number of expert demonstrations, which can be expensive or inaccessible. Learning from supplementary demonstrations without strict quality requirements has emerged as a powerful paradigm to address this challenge. However, previous methods often fail to fully utilize their potential by discarding non-expert data. Our key insight is that even demonstrations that fall outside the expert distribution but outperform the learned policy can enhance policy performance. To utilize this potential, we propose a novel approach named imitation learning via meta-learning an action ranker (ILMAR). ILMAR implements weighted behavior cloning (weighted BC) on a limited set of expert demonstrations along with supplementary demonstrations. It utilizes the functional of the advantage function to selectively integrate knowledge from the supplementary demonstrations. To make more effective use of supplementary demonstrations, we introduce meta-goal in ILMAR to optimize the functional of the advantage function by explicitly minimizing the distance between the current policy and the expert policy. Comprehensive experiments using extensive tasks demonstrate that ILMAR significantly outperforms previous methods in handling suboptimal demonstrations. Code is available at https://github.com/F-GOD6/ILMAR.

Figures (13)

• Figure 1: Left: Weighted imitation learning based on the expert distribution. Right: Weighted imitation learning based on the advantage function. Imitation learning weighted by the expert distribution fails to update the policy when non-expert demonstrations exceed the learned policy, wasting valuable data. Conversely, weighting by the advantage function recognizes superior non-expert actions, optimizing the policy and enhancing performance.
• Figure 2: Illustration of our proposed ILMAR framework. Blue dots represent demonstrations outperforming the learned policy, while red dots represent those underperforming it. The intensity of blue indicates the weight assigned by the discriminator. The discriminator reweights the supplementary dataset $\mathcal{D}^S$, filtering out inferior demonstrations. The policy is then trained on the reweighted demonstrations via behavioral cloning. After updates, the gap between the updated policy and expert demonstrations in $\mathcal{D}^E$ is used to compute the meta loss. Additionally, the relative performance of expert demonstrations, suboptimal demonstrations, random policy, and the current policy determine the vanilla loss. The discriminator is updated based on the meta loss and the vanilla loss to improve weighting.
• Figure 3: The kernel density estimates (KDE) of the log-likelihood for the expert demonstrations and the suboptimal datasets on Ant-v2 of T1, T2, and T3, based on a variational autoencoder (VAE) model.
• Figure 4: Training curves of ILMAR and baseline algorithms on tasks T1, T2, T3. The y-axis represents the normalized scores of the algorithm during training. The solid line corresponds to the average performance under five random seeds, and the shaded area corresponds to the 95% confidence interval.
• Figure 5: Training curves of expert agents on 4 locomotion control environments.

Theorems & Definitions (3)

• Theorem 1
• proof
• proof