HyperAgent: Leveraging Hypergraphs for Topology Optimization in Multi-Agent Communication

TL;DR

This work proposes HyperAgent, a hypergraph-based framework that optimizes communication topologies and effectively captures group collaboration patterns using direct hyperedge representations and incorporates a variational autoencoder framework with sparsity regularization to dynamically adjust hypergraph topologies based on task complexity.

Abstract

Recent advances in large language model-powered multi-agent systems have demonstrated remarkable collective intelligence through effective communication. However, existing approaches face two primary challenges: (i) \textit{Ineffective group collaboration modeling}, as they rely on pairwise edge representations in graph structures, limiting their ability to capture relationships among multiple agents; and (ii) \textit{Limited task-adaptiveness in communication topology design}, leading to excessive communication cost for simple tasks and insufficient coordination for complex scenarios. These issues restrict the scalability and practical deployment of adaptive collaboration frameworks. To address these challenges, we propose \textbf{HyperAgent}, a hypergraph-based framework that optimizes communication topologies and effectively captures group collaboration patterns using direct hyperedge representations. Unlike edge-based approaches, HyperAgent uses hyperedges to link multiple agents within the same subtask and employs hypergraph convolutional layers to achieve one-step information aggregation in collaboration groups. Additionally, it incorporates a variational autoencoder framework with sparsity regularization to dynamically adjust hypergraph topologies based on task complexity. Experiments highlight the superiority of HyperAgent in both performance and efficiency. For instance, on GSM8K, HyperAgent achieves 95.07\% accuracy while reducing token consumption by 25.33\%, demonstrating the potential of hypergraph-based optimization for multi-agent communication.

Figures (5)

• Figure 1: Comparison of communication topologies for multi-agent collaboration. (a) Pairwise edges require multiple connections and multi-hop information propagation among agents. (b) Hyperedges enable direct one-step synchronization by connecting all collaborating agents within a single structure.
• Figure 2: The HyperAgent pipeline. We encode agents and tasks into a hypergraph, then apply a variational autoencoder with sparsity regularization to generate task-adaptive communication topologies. Agents interact through hyperedge-based collaboration for multiple rounds. The VAE is trained via policy gradients to maximize task performance while minimizing communication overhead.
• Figure 3: Training dynamics of HyperAgent. (a) Loss components over training iterations. The utility loss (blue) steadily decreases while sparsity regularization (green) maintains stable constraint. (b) Validation accuracy improves and plateaus after 50 iterations. (c) Generated hypergraphs become progressively sparser during training, demonstrating the model learns efficient topologies.
• Figure 4: (a) Effect of hyperedge size parameter k on performance and communication efficiency. (b) Impact of sparsity regularization coefficient on the performance-efficiency frontier. (c) Information propagation: graphs need multi-hop passing whereas hyperedges enable direct 1-step synchronization. (d) Visualization of the performance metrics and prompt token consumption.
• Figure 5: Performance vs. number of interaction rounds K. Accuracy improves with more rounds but exhibits diminishing returns after K=3.