Revisiting Communication Efficiency in Multi-Agent Reinforcement Learning from the Dimensional Analysis Perspective

TL;DR

This work addresses the inefficiency of communication in multi-agent reinforcement learning by introducing dimensional analysis. DRMAC combines a redundancy-reduction objective $\mathcal{L}_{RR}$ to decorrelate embedded message dimensions with a learnable Information Selective Network (ISN) that applies a dimensional mask to emphasize decision-relevant information, the latter trained via meta-learning with second-order gradients. The method is designed to be plug-and-play, improving existing MARL baselines (e.g., MASIA, SMS, TarMAC) and even non-communication baselines when combined, across Hallway and StarCraft II SMAC environments. Empirically, DRMAC reduces dimensional redundancy and suppresses confounders, yielding superior performance and robustness in diverse, complex tasks, with strong generalizability to various baselines and settings. This approach provides a practical pathway to more efficient inter-agent communication by focusing on the information architecture of embeddings at the receiving end, rather than solely on the sender side.

Abstract

In this work, we introduce a novel perspective, i.e., dimensional analysis, to address the challenge of communication efficiency in Multi-Agent Reinforcement Learning (MARL). Our findings reveal that simply optimizing the content and timing of communication at sending end is insufficient to fully resolve communication efficiency issues. Even after applying optimized and gated messages, dimensional redundancy and confounders still persist in the integrated message embeddings at receiving end, which negatively impact communication quality and decision-making. To address these challenges, we propose Dimensional Rational Multi-Agent Communication (DRMAC), designed to mitigate both dimensional redundancy and confounders in MARL. DRMAC incorporates a redundancy-reduction regularization term to encourage the decoupling of information across dimensions within the learned representations of integrated messages. Additionally, we introduce a dimensional mask that dynamically adjusts gradient weights during training to eliminate the influence of decision-irrelevant dimensions. We evaluate DRMAC across a diverse set of multi-agent tasks, demonstrating its superior performance over existing state-of-the-art methods in complex scenarios. Furthermore, the plug-and-play nature of DRMAC's key modules highlights its generalizable performance, serving as a valuable complement rather than a replacement for existing multi-agent communication strategies.

Figures (8)

• Figure 1: Existing communication methods typically focus on Step 1 and Step 2, optimizing the content, timing, and selection of partners for communication, with the aim of enhancing efficiency at the message level. However, in this work, we observe that even after these optimizations, dimensional redundancy and confounders still exist in the integrated message embeddings at Step 3. To further improve communication efficiency, we introduce a novel perspective—dimensional analysis—as a complement to existing communication methods.
• Figure 2: The visualizations depict the representations learned by SMSsms, MASIAmasia and our method on a challenging SMAC task, 1o_2r_vs_4r. The learned features are projected into an RGB color image, where distinct colors indicate different feature types. The horizontal axis corresponds to feature dimensions, while the vertical axis represents samples from different trajectories. Greater color contrast signifies lower similarity between feature dimensions. These plots illustrate the similarity between dimensional features within a batch. In contrast to existing approaches, our method employs a redundancy-reduction technique that efficiently decouples the received information into distinct dimensions. Each of these dimensions represents a unique part of the information's entropy, ensuring that the resulting representation is both more informative and less redundant.
• Figure 3: Experimental scatter plots were generated by MASIA and SMS with randomly masked dimensions on a challenging SMAC task, 1o_2r_vs_4r. The Baseline and the red dashed lines indicate the performance achieved by the unmasked representation of MASIA and SMS. Each point represents an independent experimental result, obtained by applying a specific mask rate to the original representation at the dimensional level. Notably, the original representation remains unchanged throughout the experiments. These results demonstrate the pervasive presence of dimensional confounders in the process of multi-agent communication.
• Figure 4: The overview of our proposed DRMAC.
• Figure 5: Multiple environments considered in our experiments.