Sparse Threats, Focused Defense: Criticality-Aware Robust Reinforcement Learning for Safe Autonomous Driving

TL;DR

This work tackles robustness for DRL-based autonomous driving under sparse, safety-critical perturbations by introducing CARRL, a criticality-aware robust RL framework. It forms a general-sum Markov game between a risk exposure adversary (REA) and a risk-targeted robust agent (RTRA), enabling focused attacks on safety-critical moments while balancing safety with efficiency. The REA employs decoupled optimization for attack timing and content, supported by an adversary-guided perturbation generator, whereas the RTRA uses a Soft Actor-Critic backbone with a dual replay buffer and consistency-constrained policy optimization to learn stable, robust driving under attack. Empirical results show CARRL reduces collision rates by at least 22.66% across scenarios, with strong generalization to continuous attacks and varying traffic densities, demonstrating practical robustness improvements for safe autonomous driving.

Abstract

Reinforcement learning (RL) has shown considerable potential in autonomous driving (AD), yet its vulnerability to perturbations remains a critical barrier to real-world deployment. As a primary countermeasure, adversarial training improves policy robustness by training the AD agent in the presence of an adversary that deliberately introduces perturbations. Existing approaches typically model the interaction as a zero-sum game with continuous attacks. However, such designs overlook the inherent asymmetry between the agent and the adversary and then fail to reflect the sparsity of safety-critical risks, rendering the achieved robustness inadequate for practical AD scenarios. To address these limitations, we introduce criticality-aware robust RL (CARRL), a novel adversarial training approach for handling sparse, safety-critical risks in autonomous driving. CARRL consists of two interacting components: a risk exposure adversary (REA) and a risk-targeted robust agent (RTRA). We model the interaction between the REA and RTRA as a general-sum game, allowing the REA to focus on exposing safety-critical failures (e.g., collisions) while the RTRA learns to balance safety with driving efficiency. The REA employs a decoupled optimization mechanism to better identify and exploit sparse safety-critical moments under a constrained budget. However, such focused attacks inevitably result in a scarcity of adversarial data. The RTRA copes with this scarcity by jointly leveraging benign and adversarial experiences via a dual replay buffer and enforces policy consistency under perturbations to stabilize behavior. Experimental results demonstrate that our approach reduces the collision rate by at least 22.66\% across all cases compared to state-of-the-art baseline methods.

Figures (4)

• Figure 1: Conceptual comparison highlighting the advantages of CARRL over existing paradigms.
• Figure 2: The system model of CARRL, which consists of two components, namely, REA and RTRA. Under a limited attack budget, the REA learns to trigger perturbations at safety-critical moments, while the RTRA learns a robust driving policy by jointly leveraging benign and adversarial experiences.
• Figure 3: Performance comparison under different attack policies.
• Figure 4: Success rates of different methods under adversarial attacks across varying traffic densities and perturbation magnitudes. The top row corresponds to low-density traffic (Flow-1, $\rho = 0.3$), while the bottom row represents high-density traffic (Flow-2, $\rho = 0.7$). Each column corresponds to a different perturbation magnitude, with $\epsilon_{\text{AG}} = 0.03$, $0.05$, and $0.07$ from left to right.