Group-Aware Coordination Graph for Multi-Agent Reinforcement Learning

TL;DR

This paper addresses the challenge of coordinating multiple agents under partial observability by learning a latent coordination graph that captures both pairwise interactions and higher-order group dependencies. It introduces the Group-Aware Coordination Graph (GACG) with Gaussian-distributed edges and a group distance loss to promote cohesion within groups and specialization across groups, integrating graph-based communication into a QMIX-based MARL framework. The key contributions include a novel edge representation and sampling mechanism, a group-aware learning objective, and comprehensive ablations validating each component. Experimental results on StarCraft II micromanagement tasks show superior performance and faster convergence compared to state-of-the-art CG-based methods, with insights into distribution choices, grouping, and trajectory length influencing effectiveness.

Abstract

Cooperative Multi-Agent Reinforcement Learning (MARL) necessitates seamless collaboration among agents, often represented by an underlying relation graph. Existing methods for learning this graph primarily focus on agent-pair relations, neglecting higher-order relationships. While several approaches attempt to extend cooperation modelling to encompass behaviour similarities within groups, they commonly fall short in concurrently learning the latent graph, thereby constraining the information exchange among partially observed agents. To overcome these limitations, we present a novel approach to infer the Group-Aware Coordination Graph (GACG), which is designed to capture both the cooperation between agent pairs based on current observations and group-level dependencies from behaviour patterns observed across trajectories. This graph is further used in graph convolution for information exchange between agents during decision-making. To further ensure behavioural consistency among agents within the same group, we introduce a group distance loss, which promotes group cohesion and encourages specialization between groups. Our evaluations, conducted on StarCraft II micromanagement tasks, demonstrate GACG's superior performance. An ablation study further provides experimental evidence of the effectiveness of each component of our method.

Figures (7)

• Figure 1: In a multi-agent environment, agents may exhibit diverse behaviours represented by triangles and circles. Existing methods for modelling agent interactions primarily focus on agent-pair relations. Concurrently recognizing the importance of higher-order group relationships among agents in coordination graphs is critical.
• Figure 2: The framework of our method. GACG is designed to calculate cooperation needs between agent pairs based on current observations and to capture group-level dependencies from behaviour patterns observed across trajectories. All edges in the coordination graph are represented by a Gaussian distribution. This graph helps agents exchange knowledge when making decisions. During agent training, the group distance loss regularizes behaviour among agents with similar observation trajectories.
• Figure 3: Performance of GACG and baselines on six maps of the SMAC. The x-axis represents the time steps (in millions), while the y-axis quantifies the test win rate in the games..
• Figure 4: Experiment of choosing different edge distributions when learning the CG.
• Figure 5: Experiment of training GACG with/without $\mathcal{L}_g$.

Theorems & Definitions (3)

• Definition 1
• Definition 2
• Definition 3