Towards Fault Tolerance in Multi-Agent Reinforcement Learning

TL;DR

An attention mechanism is incorporated into the actor and critic networks to effectively and automatically detect fault information and dynamically regulate the attention given to faulty agents, and a prioritization sampling strategy is employed to select critical samples from collected experiences that are most relevant to current training needs.

Abstract

Agent faults pose a significant threat to the performance of multi-agent reinforcement learning (MARL) algorithms, introducing two key challenges. First, agents often struggle to extract critical information from the chaotic state space created by unexpected faults. Second, transitions recorded before and after faults in the replay buffer affect training unevenly, leading to a sample imbalance problem. To overcome these challenges, this paper enhances the fault tolerance of MARL by combining optimized model architecture with a tailored training data sampling strategy. Specifically, an attention mechanism is incorporated into the actor and critic networks to automatically detect faults and dynamically regulate the attention given to faulty agents. Additionally, a prioritization mechanism is introduced to selectively sample transitions critical to current training needs. To further support research in this area, we design and open-source a highly decoupled code platform for fault-tolerant MARL, aimed at improving the efficiency of studying related problems. Experimental results demonstrate the effectiveness of our method in handling various types of faults, faults occurring in any agent, and faults arising at random times.

Figures (13)

• Figure 1: (a) An illustration of a predator-prey system before and after the agent fault, where different shades of blue circles represent predators, and the green circle represents the prey. The large transparent circle around each agent represents its communication range. Following the fault of agent 2, which initially serves as a communication bridge, agent 1 and 3 suffer a loss of communication. (b) An illustration of the inputs for the actor and critic before and after fault. When agent 2 fails, its associated information becomes abnormal and is marked in red, leading to a disruption in the original input of the actor and critic networks. (c) An illustration of two natural ideas of handling faults. The left part illustrates the idea of manually distinguishing the training data and the networks before and after the agent fault, and the right part illustrates the idea of identifying the invalid information within the input automatically by the neural network. (d) An illustration of a replay buffer with transitions in 3 episodes. Blue circles represent similar pre-fault transitions and different imcomplete circles represent various post-fault transitions, reflecting imbalance of samples.
• Figure 2: An illustration of the main components of our method. Actor of agent outputs action $a_{i}$ under the continuous policy $\mu_{\theta_i}$ and critic $i$ outputs $Q_i^{\phi}(o,a)$. Information related to the faulty agent $j$ is marked in red, and the red dashed lines represent the special attention weight for the embedding of the faulty agent.
• Figure 3: The overall framework of FTMAL.
• Figure 4: Schematic diagram of the scenarios before and after faults. (a) Abandonment scenario and recovery scenario if agent 2 fails. (b) Recovery scenario if agent 3 fails. (c) Navigation scenario. (d) Patrol scenario.
• Figure 5: Testing of the basic MADDPG without considering faults in the training process. (Left) No fault. (Right) Agent 2 fails.