Multi-Agent Imitation Learning: Value is Easy, Regret is Hard

TL;DR

This work addresses how to coordinate a group of strategic agents in multi-agent imitation learning (MAIL) from demonstrations by focusing on the regret gap rather than the traditional value gap. It establishes that regret is hard to minimize in MAIL due to counterfactual deviations and coverage issues, and shows that value equivalence does not guarantee low regret. The authors introduce two efficient reductions, MALICE and BLADES, that minimize the regret gap under either full demonstration coverage or access to a queryable expert, with bounds scaling as $O( ext{epsilon} u H)$ and independent of the coverage constant in MALICE. These results highlight the necessity of accounting for counterfactual expert behavior in MAIL and provide principled pathways for robust coordination, with potential practical impact in routing, traffic management, and other coordinated multi-agent settings.

Abstract

We study a multi-agent imitation learning (MAIL) problem where we take the perspective of a learner attempting to coordinate a group of agents based on demonstrations of an expert doing so. Most prior work in MAIL essentially reduces the problem to matching the behavior of the expert within the support of the demonstrations. While doing so is sufficient to drive the value gap between the learner and the expert to zero under the assumption that agents are non-strategic, it does not guarantee robustness to deviations by strategic agents. Intuitively, this is because strategic deviations can depend on a counterfactual quantity: the coordinator's recommendations outside of the state distribution their recommendations induce. In response, we initiate the study of an alternative objective for MAIL in Markov Games we term the regret gap that explicitly accounts for potential deviations by agents in the group. We first perform an in-depth exploration of the relationship between the value and regret gaps. First, we show that while the value gap can be efficiently minimized via a direct extension of single-agent IL algorithms, even value equivalence can lead to an arbitrarily large regret gap. This implies that achieving regret equivalence is harder than achieving value equivalence in MAIL. We then provide a pair of efficient reductions to no-regret online convex optimization that are capable of minimizing the regret gap (a) under a coverage assumption on the expert (MALICE) or (b) with access to a queryable expert (BLADES).

Figures (5)

• Figure 1: Under expressive enough reward function and deviation classes, regret equivalence implies value equivalence but not vice versa, making the regret gap a "stronger" objective than the value gap.
• Figure 2: Illustration of an Markov Game that captures why "regret is hard". Here, $\sigma_E(a_1a_1|s_0)=1$. Observe that $s_1$ is un-visited when all agents obediently follow $\sigma_E$ but is with probability $1$ under deviation $\phi_1$ ($\phi_1(s_0,a_1)=\phi_1(s_1,a_1)=a_2$). This means that unless we know what the expert $\sigma_E$ would have recommended counter-factually in $s_1$, we cannot minimize the regret gap.
• Figure 3: Example of $\Omega(\frac{1}{\beta}\epsilon u H)$ regret gap for J-BC and J-IRL
• Figure 4: Example of $\Omega(\epsilon u H)$ regret gap for MALICE and BLADES
• Figure 5: Multiple reward functions rationalize $\sigma_E$

Theorems & Definitions (41)

• Definition 3.1: Regret and CE in General-Sum MGs
• Definition 4.1: Value Gap
• Definition 4.2: Regret Gap
• Theorem 4.1: Complete Classes
• Theorem 4.2: Incomplete Classes
• Theorem 4.3
• Remark 4.1
• Theorem 4.4: Regret Gap Implies CE
• Corollary 4.5
• Theorem 4.6: J-BC Value Gap Upper Bound