Towards Efficient Collaboration via Graph Modeling in Reinforcement Learning

TL;DR

The paper tackles efficient collaboration in multi-agent reinforcement learning under CTDE by introducing Factor-based Multi-Agent Transformer (f-MAT), which uses a factor graph to enable neighborhood-level message passing and parallel action generation. By representing groups of agents as factors and applying factor-based attention, f-MAT reduces communication and computation to $O(m S_f L)$ while maintaining rich inter-agent coordination. Empirical results across GridSim, Monaco traffic, and power grid scenarios show that f-MAT achieves near-MAT performance with substantially better training efficiency and scalability, highlighting its potential for large-scale, real-time cooperative systems. This work demonstrates a practical, transformer-based approach to scalable coordination in distributed multi-agent environments.

Abstract

In multi-agent reinforcement learning, a commonly considered paradigm is centralized training with decentralized execution. However, in this framework, decentralized execution restricts the development of coordinated policies due to the local observation limitation. In this paper, we consider the cooperation among neighboring agents during execution and formulate their interactions as a graph. Thus, we introduce a novel encoder-decoder architecture named Factor-based Multi-Agent Transformer ($f$-MAT) that utilizes a transformer to enable communication between neighboring agents during both training and execution. By dividing agents into different overlapping groups and representing each group with a factor, $f$-MAT achieves efficient message passing and parallel action generation through factor-based attention layers. Empirical results in networked systems such as traffic scheduling and power control demonstrate that $f$-MAT achieves superior performance compared to strong baselines, thereby paving the way for handling complex collaborative problems.

Figures (10)

• Figure 1: Factor-based attention layer. Green represents nodes, Red represents factors. (a) Factor graph: divide all nodes (1, 2, 3, 4) into two overlapping groups (1, 2, 3) and (2, 4); define two hypernodes, factor 5 and factor 6 to represent each group; transform a general graph to a bipartite graph. (b) Attention details when updating factors: to update factor observation $o_5, o_6$, we set $o_5, o_6$ as query, $o_1 \ldots o_4$ as key and value. Query $o^q_5$ only take attention to related agents' observations $o_1, o_2, o_3$. Similar operation to factor observation $o_6$. $\tilde{o}_5$ and $\tilde{o}_6$ are updated factors. (c) Update scheme on factor-based MHA: It is a two-way message passing, which first updates factors and keep nodes unchanged, then update nodes and keep factors unchanged.
• Figure 2: Architecture of $f$-MAT. At each time step, the encoder takes the observation of nodes and factors as the input and outputs the factor-based observation representation. The factor observation is initialized by the average of related agent's observations. In decoder, we initialized the actions by the learned observation representation and generate actions in parallel. All attention layers utilized are factor-based attention layers. The pseudo code of $f$-MAT can be found in Appendix \ref{['sec:pseudo']}.
• Figure 3: The performance results for GridSim with three different number of agents ($n=64, 100, 144$) and the training efficiency on grid $8 \times 8$. All performance results are presented as mean $\pm$ std. 'Efficiency' in subplot (d) refers to training efficiency, i.e., evaluation reward vs. training time.
• Figure 4: The performance and training efficiency results for traffic light control, an area in Monaco with 28 traffic lights. $f$-MAT is compatible with heterogeneous environments.
• Figure 5: The performance and training efficiency results for power grid control. $f$-MAT outperforms more evidently in complex environments.