Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrations

TL;DR

This work addresses the challenge of learning from diverse, multi-modal human demonstrations by introducing D3IL, a benchmark suite with tasks T1–T5 in MuJoCo that require multi-step manipulation and closed-loop feedback. It defines tractable diversity metrics based on behavior entropy and conditional behavior entropy to quantify multi-modality in learned policies, and benchmarks a broad set of imitation learning methods, including state- and image-based, deterministic and diffusion-based, with and without history. The study reveals that transformer- and diffusion-based architectures excel at capturing diverse behaviors, especially when equipped with history or future-action prediction, while data efficiency varies across methods. Overall, D3IL provides a rigorous framework and concrete insights to guide the development of robust, diverse imitation learning algorithms that better model human-driven multi-modal strategies.

Abstract

Imitation learning with human data has demonstrated remarkable success in teaching robots in a wide range of skills. However, the inherent diversity in human behavior leads to the emergence of multi-modal data distributions, thereby presenting a formidable challenge for existing imitation learning algorithms. Quantifying a model's capacity to capture and replicate this diversity effectively is still an open problem. In this work, we introduce simulation benchmark environments and the corresponding Datasets with Diverse human Demonstrations for Imitation Learning (D3IL), designed explicitly to evaluate a model's ability to learn multi-modal behavior. Our environments are designed to involve multiple sub-tasks that need to be solved, consider manipulation of multiple objects which increases the diversity of the behavior and can only be solved by policies that rely on closed loop sensory feedback. Other available datasets are missing at least one of these challenging properties. To address the challenge of diversity quantification, we introduce tractable metrics that provide valuable insights into a model's ability to acquire and reproduce diverse behaviors. These metrics offer a practical means to assess the robustness and versatility of imitation learning algorithms. Furthermore, we conduct a thorough evaluation of state-of-the-art methods on the proposed task suite. This evaluation serves as a benchmark for assessing their capability to learn diverse behaviors. Our findings shed light on the effectiveness of these methods in tackling the intricate problem of capturing and generalizing multi-modal human behaviors, offering a valuable reference for the design of future imitation learning algorithms.

Figures (18)

• Figure 1: DDPM-ACT
• Figure 2: GPT-based policies
• Figure 4: Ablation study for different percentages of the original dataset size of the Aligning (T2) task. The percentages are color-coded according to the legend in the top left corner.
• Figure 5: D3IL Visualizations. This figure provides an overview of various tasks and behaviors within our dataset. The top row demonstrates one of the 24 possible solutions for the "Avoiding" task. The second row displays snapshots from all four pushing sequences in the "Pushing" task, with sub-captions indicating block movements (e.g., 'rr-gg' signifies red block to red target and green block to green target). The third row showcases the two behaviors for the "Aligning" task, with the leftmost figures illustrating alignment from within the box and the rightmost from outside. The fourth row focuses on the "Sorting-3" task, with the initial configuration on the left and diverse pushing strategies, including simultaneous block manipulation, in subsequent figures. The bottom row depicts snapshots of the "Stacking" task, highlighting the intricate dexterity required, including complex pose estimation and orientation changes when picking and stacking the blue block.
• Figure 6: Initial State Space. The initial state ${{\mathbf{s}}}_0$ consists of the box and target position. The former is sampled uniformly from the yellow rectangle and the latter from the grey. The figure maintains the true-to-scale ratio of the samples and the bounding box.