Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hierarchical Imitation Learning of Team Behavior from Heterogeneous Demonstrations

Sangwon Seo, Vaibhav Unhelkar

TL;DR

DTIL addresses the challenge of learning multimodal team behavior from heterogeneous demonstrations under partial observability. It extends distribution-matching imitation learning to a multi-agent hierarchical setting by introducing a factored occupancy-measure objective that couples per-agent low and high level policies through oaxx and oaxx occupancy terms. The method employs an EM-based training loop with MAP-based subtask inference (E-step) and IQ-Learn driven policy updates (M-step), backed by theory that extends bijection and convergence results to the partially observable multi-agent regime. Empirically, DTIL demonstrates expert-level task performance, accurate multimodal behavior modeling, and superior scalability relative to BTIL and other baselines across diverse domains. This yields a practical, scalable approach for learning generative models of team behavior with potential for human-AI teaming and coaching applications.

Abstract

Successful collaboration requires team members to stay aligned, especially in complex sequential tasks. Team members must dynamically coordinate which subtasks to perform and in what order. However, real-world constraints like partial observability and limited communication bandwidth often lead to suboptimal collaboration. Even among expert teams, the same task can be executed in multiple ways. To develop multi-agent systems and human-AI teams for such tasks, we are interested in data-driven learning of multimodal team behaviors. Multi-Agent Imitation Learning (MAIL) provides a promising framework for data-driven learning of team behavior from demonstrations, but existing methods struggle with heterogeneous demonstrations, as they assume that all demonstrations originate from a single team policy. Hence, in this work, we introduce DTIL: a hierarchical MAIL algorithm designed to learn multimodal team behaviors in complex sequential tasks. DTIL represents each team member with a hierarchical policy and learns these policies from heterogeneous team demonstrations in a factored manner. By employing a distribution-matching approach, DTIL mitigates compounding errors and scales effectively to long horizons and continuous state representations. Experimental results show that DTIL outperforms MAIL baselines and accurately models team behavior across a variety of collaborative scenarios.

Hierarchical Imitation Learning of Team Behavior from Heterogeneous Demonstrations

TL;DR

DTIL addresses the challenge of learning multimodal team behavior from heterogeneous demonstrations under partial observability. It extends distribution-matching imitation learning to a multi-agent hierarchical setting by introducing a factored occupancy-measure objective that couples per-agent low and high level policies through oaxx and oaxx occupancy terms. The method employs an EM-based training loop with MAP-based subtask inference (E-step) and IQ-Learn driven policy updates (M-step), backed by theory that extends bijection and convergence results to the partially observable multi-agent regime. Empirically, DTIL demonstrates expert-level task performance, accurate multimodal behavior modeling, and superior scalability relative to BTIL and other baselines across diverse domains. This yields a practical, scalable approach for learning generative models of team behavior with potential for human-AI teaming and coaching applications.

Abstract

Successful collaboration requires team members to stay aligned, especially in complex sequential tasks. Team members must dynamically coordinate which subtasks to perform and in what order. However, real-world constraints like partial observability and limited communication bandwidth often lead to suboptimal collaboration. Even among expert teams, the same task can be executed in multiple ways. To develop multi-agent systems and human-AI teams for such tasks, we are interested in data-driven learning of multimodal team behaviors. Multi-Agent Imitation Learning (MAIL) provides a promising framework for data-driven learning of team behavior from demonstrations, but existing methods struggle with heterogeneous demonstrations, as they assume that all demonstrations originate from a single team policy. Hence, in this work, we introduce DTIL: a hierarchical MAIL algorithm designed to learn multimodal team behaviors in complex sequential tasks. DTIL represents each team member with a hierarchical policy and learns these policies from heterogeneous team demonstrations in a factored manner. By employing a distribution-matching approach, DTIL mitigates compounding errors and scales effectively to long horizons and continuous state representations. Experimental results show that DTIL outperforms MAIL baselines and accurately models team behavior across a variety of collaborative scenarios.

Paper Structure

This paper contains 54 sections, 6 theorems, 16 equations, 7 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Imitation Learning of Multimodal Behavior
  4. Multi-agent Imitation Learning
  5. Background
  6. Imitation Learning via Distribution Matching
  7. Multi-Agent Variants.
  8. Hierarchical Variants.
  9. Model of Team Behavior
  10. Dec-POMDP
  11. Agent Markov Model (AMM)
  12. Problem Formulation
  13. Formalizing Hierarchical Multi-Agent Distribution Matching
  14. Problem Statement
  15. Learning Model of Teamwork via Factored Distribution Matching
  16. ...and 39 more sections

Key Result

Theorem 5.1

For each agent $i$, given a multi-agent task model $\mathcal{M}$ and other agents' models ${{\mathcal{N}}}_{-i}$, suppose $\rho_i$ is the oaxx- occupancy measure for a stationary agent model ${\mathcal{N}}_i = (\pi_i, \zeta_i; \mathcal{M})$ where Then, ${\mathcal{N}}_i = (\pi_i, \zeta_i; \mathcal{M})$ is the only agent model whose oaxx- occupancy measure is $\rho_i$.

Figures (7)

  • Figure 1: Motivating Example: Consider a team whose members must coordinate on the fly to complete subtasks at two conveyor belts. Each member has limited observability, perceiving only their immediate surroundings. For example, the unshaded area for the blue person in \ref{['fig: workplace partially observe']}. As shown in \ref{['fig: workplace 1']}, this task allows multiple near-optimal strategies, enabling teams to execute it in different ways based on their shared preferences. However, practical constraints -- such as partial observability -- can lead to suboptimal coordination and team performance. For instance, if multiple members gather at the same subtask location, it results in inefficient task allocation, where one subtask remains unattended while two members redundantly perform the same task \ref{['fig: workplace suboptimal']}. Like many real-world scenarios, this task engenders heterogeneous and potentially suboptimal demonstrations of teamwork. This paper focuses on learning models of team behavior in this challenging setting from demonstrations.
  • Figure 2: Snapshopts of Experimental Domains
  • Figure 3: Visualization of individual Multi-Jobs-$3$ trajectories generated by the expert and learned models conditioned on a fixed subtask. The directions of the triangles and arrows represent the actions of agents at each position. The three colors represent the three fixed subtasks. Both learned models (DTIL and MA-OptionGAIL) are trained with 20 % supervision of subtask labels.
  • Figure 4: Multi-Jobs-$3$
  • Figure 5: The average task returns vs. the number of exploration steps. Each method is plotted with three seeds.
  • ...and 2 more figures

Theorems & Definitions (6)

  • Theorem 5.1
  • Lemma 7.1
  • Theorem 7.2
  • Lemma A.3
  • Lemma A.4: Lemma 2.3 of seo2024idil
  • Corollary A.5