Learning Efficient Communication Protocols for Multi-Agent Reinforcement Learning

TL;DR

This work tackles the inefficiency of inter-agent communication in multi-agent reinforcement learning by proposing a generalized framework for multi-round, learnable communication protocols under centralized training and decentralized execution (CTDE). It introduces three high-signal communication efficiency metrics—Information Entropy Efficiency Index ($ ext{IEI}$), Specialization Efficiency Index ($ ext{SEI}$), and Topology Efficiency Index ($ ext{TEI}$)—and demonstrates how incorporating $ ext{IEI}$ and $ ext{SEI}$ into the training loss (with dynamic weighting) can yield efficient, well-coordinated communication without sacrificing task performance; $ ext{TEI}$ is used for explicit evaluation of communication costs. The approach is validated across several MARL baselines (e.g., IC3Net, CommNet, TarMAC, MAGIC, GA-Comm) in a Traffic Junction setting, showing that efficiency augmentation improves both success rates and communication parsimony, often with one-round communication outperforming or equalling multi-round setups under constrained resources. The results suggest that efficiency objectives can be optimized simultaneously with task performance, enabling practical deployment in resource-limited multi-agent systems and informing future work on generalizing these techniques to broader environments and human-agent collaborations.

Abstract

Multi-Agent Systems (MAS) have emerged as a powerful paradigm for modeling complex interactions among autonomous entities in distributed environments. In Multi-Agent Reinforcement Learning (MARL), communication enables coordination but can lead to inefficient information exchange, since agents may generate redundant or non-essential messages. While prior work has focused on boosting task performance with information exchange, the existing research lacks a thorough investigation of both the appropriate definition and the optimization of communication protocols (communication topology and message). To fill this gap, we introduce a generalized framework for learning multi-round communication protocols that are both effective and efficient. Within this framework, we propose three novel Communication Efficiency Metrics (CEMs) to guide and evaluate the learning process: the Information Entropy Efficiency Index (IEI) and Specialization Efficiency Index (SEI) for efficiency-augmented optimization, and the Topology Efficiency Index (TEI) for explicit evaluation. We integrate IEI and SEI as the adjusted loss functions to promote informative messaging and role specialization, while using TEI to quantify the trade-off between communication volume and task performance. Through comprehensive experiments, we demonstrate that our learned communication protocol can significantly enhance communication efficiency and achieves better cooperation performance with improved success rates.

Figures (5)

• Figure 1: General MARL Framework of Multi-round Communications. (a) Centralized Training, (b) Decentralized Execution.
• Figure 2: Comparison of $\Phi_{{\text{IEI}}}$, $\Phi_{{\text{SEI}}}$, and $\Phi_{{\text{TEI}}}$ for different algorithms in the TJ environment with $L=1$.
• Figure 3: Comparison of $\Phi_{{\text{IEI}}}$, $\Phi_{{\text{SEI}}}$, $\Phi_{{\text{TEI}}}$ for different algorithms under one-round and two-round communication scenario in the TJ environment.
• Figure 4: Comparison of succcess rate and $\Phi_{{\text{TEI}}}$ for different algorithms in the TJ environment of one-round communication scenario and two-round communication scenario.
• Figure 5: Comparison of (a) $\Phi_{{\text{IEI}}}$ and (b) $\Phi_{{\text{SEI}}}$ for CommNet and IC3Net in the TJ environment of one-round communication scenario.