Fault-Tolerant MARL for CAVs under Observation Perturbations for Highway On-Ramp Merging

TL;DR

This work tackles the fault-tolerance challenge of multi-agent reinforcement learning for cooperative CAVs in highway on-ramp merging by introducing OFT-MARL, a framework pairing an adversarial fault-injection agent with a fault-tolerant vehicle agent. The vehicle agent uses a GRU-based temporal discriminator to detect faults and reconstruct credible observations, while the fault injector challenges policies during training to hardened behavior. Through joint training within a MADDPG-based multi-agent setup, OFT-MARL achieves near fault-free safety and efficiency across diverse fault patterns and demonstrates robust generalization to unseen perturbations. The results suggest significant practical impact for deploying MARL in real-world autonomous driving, offering a pathway to resilient cooperative control under imperfect sensing and communication conditions.

Abstract

Multi-Agent Reinforcement Learning (MARL) holds significant promise for enabling cooperative driving among Connected and Automated Vehicles (CAVs). However, its practical application is hindered by a critical limitation, i.e., insufficient fault tolerance against observational faults. Such faults, which appear as perturbations in the vehicles' perceived data, can substantially compromise the performance of MARL-based driving systems. Addressing this problem presents two primary challenges. One is to generate adversarial perturbations that effectively stress the policy during training, and the other is to equip vehicles with the capability to mitigate the impact of corrupted observations. To overcome the challenges, we propose a fault-tolerant MARL method for cooperative on-ramp vehicles incorporating two key agents. First, an adversarial fault injection agent is co-trained to generate perturbations that actively challenge and harden the vehicle policies. Second, we design a novel fault-tolerant vehicle agent equipped with a self-diagnosis capability, which leverages the inherent spatio-temporal correlations in vehicle state sequences to detect faults and reconstruct credible observations, thereby shielding the policy from misleading inputs. Experiments in a simulated highway merging scenario demonstrate that our method significantly outperforms baseline MARL approaches, achieving near-fault-free levels of safety and efficiency under various observation fault patterns.

Figures (9)

• Figure 1: Vehicle 1 suddenly misobserves the position of vehicle 2, perceiving it at the location marked in red (Left Middle). In the fault-free scenario, all vehicles would maintain velocities and proceed according to the original cooperative strategy (Left Top). However, influenced by the fault, vehicle 1 chooses to decelerate and yield, which is contrasted with the training experience from the perspective of vehicle 3 (Left Bottom). Further, vehicle 3's action becomes unpredictable due to deviations in vehicle 1's behavior. Regardless of vehicle 3's action, traffic efficiency or safety may be affected for different reasons (Right). The blue, green, and red colors behind vehicles indicate their maintenance, acceleration and deceleration behaviors respectively.
• Figure 2: Overall framework of OFT-MARL.
• Figure 3: Schematic diagram of OFT-MARL. Top: Illustration of a on-ramp merging scenario case (left) and the composition of the observation for vehicle 1 within the scenario (right). Middle: Workflow of the adversarial fault injection agent, which takes the observations of all vehicles and two fault indicators as input, processes them through the fault generation network to output perturbation magnitude $b$, and then transforms the true observation $o_{ij}$ into fault-affected observation $f(o_{ij},b)$ via the fault-affected function. Bottom: Workflow of the fault-tolerant vehicle agent, which takes potentially perturbed observation $\hat{o}_i$ along with the hidden state $h_i$ from the previous timestep (via the GRU), outputs the hidden state $h'_i$ for the next timestep, further processes it through an MLP network to generate fault predictions $\tilde{p}_{i}$ and observation estimates $\tilde{o}_{i}$, and finally combines these with the observation $\hat{o}_i$ to produce acceleration action $a_i$ through the action output network.
• Figure 4: (a) Average reward curves for MADDPG in fault-free scenarios. The error bar is a 95% confidence interval across 6 runs. (b) Performance decline of methods trained in fault-free scenarios and tested under fault-free and random fault conditions.
• Figure 5: Influence of random faults.