HYGMA: Hypergraph Coordination Networks with Dynamic Grouping for Multi-Agent Reinforcement Learning

TL;DR

HYGMA tackles dynamic coordination in multi-agent reinforcement learning by combining dynamic spectral clustering with hypergraph neural networks. It automatically forms agent groups from state histories and processes intra- and inter-group information via an attention-based hypergraph convolution, applicable to both value-based and policy-based MARL under CTDE. The framework provides theoretical guarantees on clustering convergence and learning quality, and demonstrates superior sample efficiency and final performance across SMAC, Predator-Prey, Traffic Junction, and Google Research Football benchmarks, with meaningful reductions in communication overhead. Ablation studies confirm the critical role of the dynamic hypergraph structure in discovering sophisticated coordination patterns, while analysis reveals interpretable emergent groupings. Overall, HYGMA offers a scalable, expressive, and practical approach to adaptive higher-order coordination in cooperative MARL.

Abstract

Cooperative multi-agent reinforcement learning faces significant challenges in effectively organizing agent relationships and facilitating information exchange, particularly when agents need to adapt their coordination patterns dynamically. This paper presents a novel framework that integrates dynamic spectral clustering with hypergraph neural networks to enable adaptive group formation and efficient information processing in multi-agent systems. The proposed framework dynamically constructs and updates hypergraph structures through spectral clustering on agents' state histories, enabling higher-order relationships to emerge naturally from agent interactions. The hypergraph structure is enhanced with attention mechanisms for selective information processing, providing an expressive and efficient way to model complex agent relationships. This architecture can be implemented in both value-based and policy-based paradigms through a unified objective combining task performance with structural regularization. Extensive experiments on challenging cooperative tasks demonstrate that our method significantly outperforms state-of-the-art approaches in both sample efficiency and final performance.

Figures (5)

• Figure 1: Overview of the proposed hypergraph-based dynamic grouping multi-agent reinforcement learning framework with spectral clustering and attentive information processing.
• Figure 2: The test won rate for the 3s_vs_5z (Left), the 5m_vs_6m (Middle) and the 3s5z_vs_3s6z (Right).
• Figure 3: Average steps per episode in Predator-Prey.
• Figure 4: Group analysis in 5m_vs_6m.
• Figure 5: Ablation studies in 5m_vs_6m.

Theorems & Definitions (25)

• Theorem 3.1: Clustering Approximation
• Theorem 3.2: Convergence
• Theorem 3.3: Quality Preservation
• Theorem 1.1: Clustering Approximation
• Lemma 1.2: Discrete-Continuous Relation
• proof
• Lemma 1.3: Spectral Relaxation Bound
• proof
• Lemma 1.4: Dynamic Grouping Bound
• proof