SPECTra: Scalable Multi-Agent Reinforcement Learning with Permutation-Free Networks
Hyunwoo Park, Baekryun Seong, Sang-Ki Ko
TL;DR
The paper tackles permutation-related state explosion and scalability in cooperative MARL by introducing SPECTra, a Transformer-based framework that yields permutation-free agent and mixing networks capable of handling variable agent counts. It combines a lightweight Single-Agent Query Attention (SAQA) mechanism with a Set Transformer-based hypernetwork (ST-HyperNet) to generate weights for a permutation-invariant mixer, preserving monotonicity and enabling scalable, order-agnostic value estimation. The authors provide theoretical and empirical support, including complexity reasoning for SAQA ($O(nd)$) and extensive experiments on SMACv2 and Google Research Football demonstrating faster training, fewer parameters, and strong transfer and curriculum-learning performance. This work offers a practical, scalable approach to cooperative MARL with broad applicability to dynamic agent populations and complex multi-agent environments.
Abstract
In cooperative multi-agent reinforcement learning (MARL), the permutation problem where the state space grows exponentially with the number of agents reduces sample efficiency. Additionally, many existing architectures struggle with scalability, relying on a fixed structure tied to a specific number of agents, limiting their applicability to environments with a variable number of entities. While approaches such as graph neural networks (GNNs) and self-attention mechanisms have progressed in addressing these challenges, they have significant limitations as dense GNNs and self-attention mechanisms incur high computational costs. To overcome these limitations, we propose a novel agent network and a non-linear mixing network that ensure permutation-equivariance and scalability, allowing them to generalize to environments with various numbers of agents. Our agent network significantly reduces computational complexity, and our scalable hypernetwork enables efficient weight generation for non-linear mixing. Additionally, we introduce curriculum learning to improve training efficiency. Experiments on SMACv2 and Google Research Football (GRF) demonstrate that our approach achieves superior learning performance compared to existing methods. By addressing both permutation-invariance and scalability in MARL, our work provides a more efficient and adaptable framework for cooperative MARL. Our code is available at https://github.com/funny-rl/SPECTra.