Module-wise Adaptive Adversarial Training for End-to-end Autonomous Driving

TL;DR

This paper tackles the vulnerability of end-to-end autonomous driving models to adversarial perturbations by introducing MA2T, a defense combining Module-wise Noise Injection and Dynamic Weight Accumulation Adaptation to align adversarial training with the holistic, interdependent AD pipeline. By injecting noise at the inputs of perception, prediction, and planning modules and adaptively balancing module losses, MA2T achieves substantial improvements in robustness under both white-box and black-box attacks on nuScenes, with additional gains in closed-loop CARLA simulations including resilience to natural corruption. The authors validate MA2T across UniAD and VAD architectures, showing 5–10% absolute improvements in planning robustness and noteworthy reductions in planning error, while maintaining competitive clean performance. The work demonstrates MA2T’s practical impact for safer end-to-end AD systems and provides comprehensive ablations, adaptive attack analyses, and real-world corruption evaluations to support its effectiveness and generalizability.

Abstract

Recent advances in deep learning have markedly improved autonomous driving (AD) models, particularly end-to-end systems that integrate perception, prediction, and planning stages, achieving state-of-the-art performance. However, these models remain vulnerable to adversarial attacks, where human-imperceptible perturbations can disrupt decision-making processes. While adversarial training is an effective method for enhancing model robustness against such attacks, no prior studies have focused on its application to end-to-end AD models. In this paper, we take the first step in adversarial training for end-to-end AD models and present a novel Module-wise Adaptive Adversarial Training (MA2T). However, extending conventional adversarial training to this context is highly non-trivial, as different stages within the model have distinct objectives and are strongly interconnected. To address these challenges, MA2T first introduces Module-wise Noise Injection, which injects noise before the input of different modules, targeting training models with the guidance of overall objectives rather than each independent module loss. Additionally, we introduce Dynamic Weight Accumulation Adaptation, which incorporates accumulated weight changes to adaptively learn and adjust the loss weights of each module based on their contributions (accumulated reduction rates) for better balance and robust training. To demonstrate the efficacy of our defense, we conduct extensive experiments on the widely-used nuScenes dataset across several end-to-end AD models under both white-box and black-box attacks, where our method outperforms other baselines by large margins (+5-10%). Moreover, we validate the robustness of our defense through closed-loop evaluation in the CARLA simulation environment, showing improved resilience even against natural corruption.

Figures (8)

• Figure 1: Illustration of $\text{MA}^{\text{2}}\text{T}$ (using UniAD as an example). Noise can be introduced at different modules, either directly on the input data or within the connections between modules.
• Figure 2: Defense results under adaptive white-box attacks.
• Figure 3: The detailed loss trend with/without Dynamic Weight Accumulation Adaptation. (All losses are proportionally scaled down to the 0-1 range.)
• Figure 4: Defense results of plan's Avg. L2 Error (m) $\downarrow$ under three adaptive white-box attacks.
• Figure 5: Comparison images of the (a) vanilla model under clean conditions, (b) after attack, and (c) $\text{MA}^{\text{2}}\text{T}$ trained model after attack. The images show that the original model slows down and avoids when there is a bicycle ahead. However, after an attack, the ego vehicle directly collides with the bicycle. When using $\text{MA}^{\text{2}}\text{T}$ trained model during the attack, the ego vehicle adjusts to the left to avoid the bicycle.