Robust Multi-agent Communication Based on Decentralization-Oriented Adversarial Training

TL;DR

This paper tackles the fragility of centralized communication in multi-agent systems by introducing DMAC, a decentralization-oriented adversarial training framework for robust CP. A dedicated adversary, DMAC_Adv, dynamically masks critical communication channels and is trained to maximize an adversary reward that balances system performance with the number of masked channels, guiding CP toward decentralized communication through adversarial retraining. The approach is compatible with any learnable CP and is evaluated on four multi-agent benchmarks (SC, MPE CN/PP, TJ) with two CP baselines (T2MAC and I2C), showing substantial gains in robustness and improved decentralization without significant cost increases. Overall, DMAC provides a practical path to more resilient and scalable inter-agent communication in MARL, with code and results released to support replication and further research.

Abstract

In typical multi-agent reinforcement learning (MARL) problems, communication is important for agents to share information and make the right decisions. However, due to the complexity of training multi-agent communication, existing methods often fall into the dilemma of local optimization, which leads to the concentration of communication in a limited number of channels and presents an unbalanced structure. Such unbalanced communication policy are vulnerable to abnormal conditions, where the damage of critical communication channels can trigger the crash of the entire system. Inspired by decentralization theory in sociology, we propose DMAC, which enhances the robustness of multi-agent communication policies by retraining them into decentralized patterns. Specifically, we train an adversary DMAC\_Adv which can dynamically identify and mask the critical communication channels, and then apply the adversarial samples generated by DMAC\_Adv to the adversarial learning of the communication policy to force the policy in exploring other potential communication schemes and transition to a decentralized structure. As a training method to improve robustness, DMAC can be fused with any learnable communication policy algorithm. The experimental results in two communication policies and four multi-agent tasks demonstrate that DMAC achieves higher improvement on robustness and performance of communication policy compared with two state-of-the-art and commonly-used baselines. Also, the results demonstrate that DMAC can achieve decentralized communication structure with acceptable communication cost.

Figures (5)

• Figure 1: The communication frequency of the communication channel between each pair of agents in two environments. The darker the color, the more times the two agents communicate, that is, the higher the communication frequency of the communication channel. The grids on the diagonal represent the same agent and are therefore white.
• Figure 2: The overview of our proposed DMAC.
• Figure 3: The overview of DMAC_Adv. (a) At each time step, DMAC_Adv obtains the feature of each agent from the multi-agent environment, and outputs a decision for each communication channel whether or not to mask. (b) During the training, the masking agents' policy network learns the masking action and individual value, and the critic network learns the total value to estimate the expected reward. The loss function is introduced to minimize the difference between the expected reward and actual reward.
• Figure 4: Multiple environments used in our experiments.
• Figure 5: The communication frequency of the communication channel between each pair of agents in two environments. The darker the color, the more times the two agents communicate, that is, the higher the communication frequency of the communication channel. The grids on the diagonal represent the same agent and are therefore white.