R2BC: Multi-Agent Imitation Learning from Single-Agent Demonstrations

TL;DR

The paper addresses learning cooperative multi-agent policies when a single human cannot provide joint demonstrations for $N$ agents. It introduces Round-Robin Behavior Cloning (R2BC), an online, decentralized IL method that learns from sequential single-agent demonstrations while other agents execute learned policies. Across four simulated VMAS tasks and two physical robot tasks, R2BC matches or exceeds the performance of oracle centralized baselines and significantly outperforms them in real-world deployments, especially when modest human interventions are allowed. The work highlights the practical viability of scalable multi-agent IL with realistic human demonstration capabilities and suggests future work on user studies and convergence guarantees.

Abstract

Imitation Learning (IL) is a natural way for humans to teach robots, particularly when high-quality demonstrations are easy to obtain. While IL has been widely applied to single-robot settings, relatively few studies have addressed the extension of these methods to multi-agent systems, especially in settings where a single human must provide demonstrations to a team of collaborating robots. In this paper, we introduce and study Round-Robin Behavior Cloning (R2BC), a method that enables a single human operator to effectively train multi-robot systems through sequential, single-agent demonstrations. Our approach allows the human to teleoperate one agent at a time and incrementally teach multi-agent behavior to the entire system, without requiring demonstrations in the joint multi-agent action space. We show that R2BC methods match, and in some cases surpass, the performance of an oracle behavior cloning approach trained on privileged synchronized demonstrations across four multi-agent simulated tasks. Finally, we deploy R2BC on two physical robot tasks trained using real human demonstrations.

Figures (6)

• Figure 1: Round-Robin Behavior Cloning (R2BC): Traditional Behavior Cloning (left) requires coordinated and centralized demonstrations, where an expert demonstrates actions near-optimally for all agents. In multi-agent domains, a lone human operator may not be able to provide high-quality demonstrations due to underactuated control and increased cognitive burden. Our method (right), R2BC, removes this restriction by letting the human control one agent at a time while the other agents act via their learned policies. This round-robin process collects realistic demonstrations and iteratively trains cooperative multi-agent behavior.
• Figure 2: Simulation Results: We compare the R2BC method to a set of three baselines that assume oracle (privileged) access to a joint action demonstrator. Results are averaged over 10 seeds with shaded regions indicating a 99% confidence interval. Top: Four multi-agent tasks selected from the Vectorized Multi-Agent Simulator (VMAS) bettini2022vmas including Navigation ($N=3$), Balance ($N=3$), Buzz Wire ($N=2$), and Transport ($N=3$). Bottom: The performance of each method is shown with increasing number of expert demonstrations. In each case, R2BC outperforms or achieves parity with all baseline oracle methods, despite not having privileged access to joint action demonstrations. Performance is normalized between 0.0 (random policy) and 1.0 (oracle demonstrator performance).
• Figure 3: Covariate Shift Reduction: The Loss Gap is defined as the difference between the training loss on the demonstration set and testing loss on evaluation trajectories (lower is better). Dashed lines indicate methods that require privileged centralized demonstrations, while the solid blue line is our method. R2BC reduces the train–test loss gap compared to JBC, while showing benefits compared to other online imitation learning methods. Shaded regions indicate standard error across 10 seeds.
• Figure 4: R2BC Ablations: Comparison of R2BC to several ablations across four simulated environments. Across all environments, R2BC consistently outperforms the two offline variants and matches or exceeds the oracle JBC methods. Error bars indicate 99% confidence interval over 10 seeded runs.
• Figure 5: Real Robot Experiments:(a) The HeRo+ Robot, a differential drive mobile robot that acts using a forward velocity and angular velocity, (b) AruCo markers used to track robot/goal states, (c) R2BC demonstrations are provided in simulation by a real human, (d) a real world instantiation of the navigation task (from Figure \ref{['fig:res-main']} top), where agents must navigate to their colored goals while avoiding collisions, (e) a block-pusher task where robots must transport a 0.5kg box across the environment. In both (d) and (e), the pink AruCo marker is used for camera-to-world registration.