Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Demystifying the Physics of Deep Reinforcement Learning-Based Autonomous Vehicle Decision-Making

Hanxi Wan, Pei Li, Arpan Kusari

TL;DR

This paper probes how attention-based DRL internalizes the physics of autonomous-vehicle decision-making, proposing a framework that combines PPO-based baselines, multi-head attention, and a rigorous suite of explainability and causal analyses. Using a 7×5 affordance-feature representation and a two-stage training regime, it shows that two attention heads offer favorable performance and that attention weights encode spatial and temporal vehicle interactions. The authors demonstrate spatial and temporal explanations via attention patterns and establish causal relations with Iterative Causal Discovery around lane-change events, highlighting leader/follower dynamics in the target lane. Overall, the work advances interpretable DRL for AVs by linking learned policy signals to interpretable physical interactions, with practical implications for safety and reliability.

Abstract

With the advent of universal function approximators in the domain of reinforcement learning, the number of practical applications leveraging deep reinforcement learning (DRL) has exploded. Decision-making in autonomous vehicles (AVs) has emerged as a chief application among them, taking the sensor data or the higher-order kinematic variables as the input and providing a discrete choice or continuous control output. There has been a continuous effort to understand the black-box nature of the DRL models, but so far, there hasn't been any discussion (to the best of authors' knowledge) about how the models learn the physical process. This presents an overwhelming limitation that restricts the real-world deployment of DRL in AVs. Therefore, in this research work, we try to decode the knowledge learnt by the attention-based DRL framework about the physical process. We use a continuous proximal policy optimization-based DRL algorithm as the baseline model and add a multi-head attention framework in an open-source AV simulation environment. We provide some analytical techniques for discussing the interpretability of the trained models in terms of explainability and causality for spatial and temporal correlations. We show that the weights in the first head encode the positions of the neighboring vehicles while the second head focuses on the leader vehicle exclusively. Also, the ego vehicle's action is causally dependent on the vehicles in the target lane spatially and temporally. Through these findings, we reliably show that these techniques can help practitioners decipher the results of the DRL algorithms.

Demystifying the Physics of Deep Reinforcement Learning-Based Autonomous Vehicle Decision-Making

TL;DR

This paper probes how attention-based DRL internalizes the physics of autonomous-vehicle decision-making, proposing a framework that combines PPO-based baselines, multi-head attention, and a rigorous suite of explainability and causal analyses. Using a 7×5 affordance-feature representation and a two-stage training regime, it shows that two attention heads offer favorable performance and that attention weights encode spatial and temporal vehicle interactions. The authors demonstrate spatial and temporal explanations via attention patterns and establish causal relations with Iterative Causal Discovery around lane-change events, highlighting leader/follower dynamics in the target lane. Overall, the work advances interpretable DRL for AVs by linking learned policy signals to interpretable physical interactions, with practical implications for safety and reliability.

Abstract

With the advent of universal function approximators in the domain of reinforcement learning, the number of practical applications leveraging deep reinforcement learning (DRL) has exploded. Decision-making in autonomous vehicles (AVs) has emerged as a chief application among them, taking the sensor data or the higher-order kinematic variables as the input and providing a discrete choice or continuous control output. There has been a continuous effort to understand the black-box nature of the DRL models, but so far, there hasn't been any discussion (to the best of authors' knowledge) about how the models learn the physical process. This presents an overwhelming limitation that restricts the real-world deployment of DRL in AVs. Therefore, in this research work, we try to decode the knowledge learnt by the attention-based DRL framework about the physical process. We use a continuous proximal policy optimization-based DRL algorithm as the baseline model and add a multi-head attention framework in an open-source AV simulation environment. We provide some analytical techniques for discussing the interpretability of the trained models in terms of explainability and causality for spatial and temporal correlations. We show that the weights in the first head encode the positions of the neighboring vehicles while the second head focuses on the leader vehicle exclusively. Also, the ego vehicle's action is causally dependent on the vehicles in the target lane spatially and temporally. Through these findings, we reliably show that these techniques can help practitioners decipher the results of the DRL algorithms.
Paper Structure (24 sections, 4 equations, 7 figures)

This paper contains 24 sections, 4 equations, 7 figures.

Table of Contents

  1. Introduction
  2. Background
  3. AV Decision-making
  4. Explainability and Causality in AV Decision-making
  5. Methodology
  6. State and action representation
  7. Model architectures
  8. Baseline Model
  9. Multi-Head Attention-Based Model
  10. Reward Function
  11. Training Process
  12. Reinforcement learning
  13. Imitation Pre-Training for Attention-based framework
  14. Explainability using the attention mechanism
  15. Causality using Iterative Causal Discovery
  16. ...and 9 more sections

Figures (7)

  • Figure 1: Architecture of the attention-based feature extractor.
  • Figure 2: Comparing rewards over iterations for different heads
  • Figure 3: Baseline model vs the attention-based model using common driving metrics
  • Figure 4: 2D histogram showing the change in attention weights with respect to normalized distance to the leader vehicle during cumulative left and right lane changes - the weights can be shown to have direct correlation with distance although the rate of change differs significantly
  • Figure 5: The top figure shows the lane change of the ego vehicle shown in yellow with the neighboring vehicles with the evolution into the future. The bottom plot shows the attention weights of the two heads for the lead vehicles in the three lanes mentioned in the legend.
  • ...and 2 more figures