Multi-Agent Synchronization Tasks

TL;DR

This work defines Multi-Agent Synchronization Tasks (MSTs) as a Dec-POMDP subclass where each agent's action space is partitioned into synchronization and neutral actions, introducing high-stakes coordination via sets $\mathbb{A}^+(s)$ and $\mathbb{A}^-(s)$. It presents Synchronized Predator-Prey as a concrete MST benchmark and evaluates state-of-the-art coordination-graph methods (DCG, DICG, QGNN) across varying team sizes and action heterogeneity. Results indicate that DCG offers the strongest performance among evaluated methods but cannot scale beyond 2-agent sub-teams or handle heterogeneous capture actions, while DICG and QGNN generally fail under these MST conditions; removing the miscapture penalty eliminates the MST constraint, enabling non-MST solutions. These findings challenge the applicability of current CG/GNN-based MARL approaches to complex coordination tasks and motivate further research into scalable, representation-rich message-passing frameworks for MST-like coordination. The paper contributes a formal MST framework and a tangible benchmark to guide future development of robust coordination strategies in multi-agent systems.

Abstract

In multi-agent reinforcement learning (MARL), coordination plays a crucial role in enhancing agents' performance beyond what they could achieve through cooperation alone. The interdependence of agents' actions, coupled with the need for communication, leads to a domain where effective coordination is crucial. In this paper, we introduce and define $\textit{Multi-Agent Synchronization Tasks}$ (MSTs), a novel subset of multi-agent tasks. We describe one MST, that we call $\textit{Synchronized Predator-Prey}$, offering a detailed description that will serve as the basis for evaluating a selection of recent state-of-the-art (SOTA) MARL algorithms explicitly designed to address coordination challenges through the use of communication strategies. Furthermore, we present empirical evidence that reveals the limitations of the algorithms assessed to solve MSTs, demonstrating their inability to scale effectively beyond 2-agent coordination tasks in scenarios where communication is a requisite component. Finally, the results raise questions about the applicability of recent SOTA approaches for complex coordination tasks (i.e. MSTs) and prompt further exploration into the underlying causes of their limitations in this context.

Figures (5)

• Figure 1: Synchronized Predator-Prey Task. Blue arrows denote movement (neutral) actions and purple arrows denote capture (synchronization) actions.
• Figure 2: Visual representation of the payoff relationship in the Synchronized Predator-Prey Task for 2-agent sub-teams. Of note are the strict inequalities between the possible $Q$-values.
• Figure 3: Train episode reward (Mean and shaded Standard Deviation) for ten independently trained models. The Full CG topology was used, except for DICG which used an Attention mechanism to create the graph. Note that SOTA methods did not solve MSTs well. Currently, DCG is the best solution (i.e., (a) and (b)) but cannot scale to larger sub-teams or handle more complex coordination (i.e., (c) and (d)).
• Figure 4: Train episode reward (Mean and shaded Standard Deviation) for ten independently trained models. The Full CG topology was used for (a) and (b), except for DICG which used an Attention mechanism to create the graph. For (c) and (d), the Empty CG topology was used for all algorithms. Note that communication is necessary for MSTs but is not sufficient when complexity increases.
• Figure 5: Train episode reward (Mean and shaded Standard Deviation) for ten independently trained models. The miscapture penalty was disabled for these training iterations. The Full CG topology was used, except for DICG which used an Attention mechanism to create the graph. Note with the miscapture penalty disabled the task no longer satisfies the requirements for an MST and was solved by all SOTA methods.

Theorems & Definitions (1)

• Definition 1