Bridging Training and Execution via Dynamic Directed Graph-Based Communication in Cooperative Multi-Agent Systems

TL;DR

This work tackles the reliance on global state in centralized training for cooperative multi-agent reinforcement learning by introducing TGCNet, which models communication as dynamic directed graphs and bridges training and execution. It combines a graph coarsening network to approximate global state with a Transformer-based multi-key gated communication mechanism, enabling selective, bidirectional information exchange without requiring global state inputs. Theoretical analysis and architectural design support end-to-end training, while extensive experiments on Hallway, Level-Based Foraging, and SMAC demonstrate state-of-the-art performance and robust ablations. The approach yields interpretable communication patterns and scalable coordination, with potential impact on real-world multi-agent systems where global state is unavailable or costly to obtain.

Abstract

Multi-agent systems must learn to communicate and understand interactions between agents to achieve cooperative goals in partially observed tasks. However, existing approaches lack a dynamic directed communication mechanism and rely on global states, thus diminishing the role of communication in centralized training. Thus, we propose the Transformer-based graph coarsening network (TGCNet), a novel multi-agent reinforcement learning (MARL) algorithm. TGCNet learns the topological structure of a dynamic directed graph to represent the communication policy and integrates graph coarsening networks to approximate the representation of global state during training. It also utilizes the Transformer decoder for feature extraction during execution. Experiments on multiple cooperative MARL benchmarks demonstrate state-of-the-art performance compared to popular MARL algorithms. Further ablation studies validate the effectiveness of our dynamic directed graph communication mechanism and graph coarsening networks.

Figures (7)

• Figure 1: Dynamic directed graph. (a) Adjacency trajectory matrix at time $t$. (b) Dynamic directed graph. The dynamic directed graph can represent the associations and communication structures between nodes at a certain moment.
• Figure 2: The network structure diagram of TGCNet. (a) Graph coarsening network and mix network. The inputs for the graph coarsening network include the local observations of the agents and the adjacency trajectory matrix. (b) Overall TGCNet architecture. (c) Transformer-Based multi-key gated communication mechanism. The communication mechanism outputs not only the individual state value function $Q^i(\tau^i,a^i)$ and hidden variables $h_t^i$ but also the adjacency trajectory matrix $A^i_{t}$. Then, this matrix is passed into the graph coarsening network to complete end-to-end backward propagation updates.
• Figure 3: Multiple benchmarks used in our experiments.
• Figure 4: Performance comparison with baselines on multiple benchmarks.
• Figure 5: Performance comparison with baselines on hard (first row) and super hard (second row) scenarios in SMAC.