Multilevel Graph Reinforcement Learning for Consistent Cognitive Decision-making in Heterogeneous Mixed Autonomy

TL;DR

The paper tackles decision-making for heterogeneous mixed autonomy by modeling spatiotemporal vehicle interactions in non-Euclidean spaces using a multilevel graph representation. It jointly leverages an asynchronous parallel hierarchical graph reinforcement learning framework (PAH-GRL) and a Multilevel Multi-head Graph Attention Network (ML-MGAT) to learn coordinated lane-changing and following policies across dimensions L and F. Key contributions include (i) a formal AMG-MDP formulation with dimension-specific state/action/reward definitions, (ii) dynamic, multidimensional, and weighted traffic graphs to capture complex interactions, and (iii) empirical demonstrations showing substantial improvements in task success, braking events, travel time, energy, and trajectory robustness on a high-density highway scenario. The approach advances scalable, human-like cognitive decision-making in CAVs with potential practical impact for safer, more efficient mixed-traffic systems.

Abstract

In the realm of heterogeneous mixed autonomy, vehicles experience dynamic spatial correlations and nonlinear temporal interactions in a complex, non-Euclidean space. These complexities pose significant challenges to traditional decision-making frameworks. Addressing this, we propose a hierarchical reinforcement learning framework integrated with multilevel graph representations, which effectively comprehends and models the spatiotemporal interactions among vehicles navigating through uncertain traffic conditions with varying decision-making systems. Rooted in multilevel graph representation theory, our approach encapsulates spatiotemporal relationships inherent in non-Euclidean spaces. A weighted graph represents spatiotemporal features between nodes, addressing the degree imbalance inherent in dynamic graphs. We integrate asynchronous parallel hierarchical reinforcement learning with a multilevel graph representation and a multi-head attention mechanism, which enables connected autonomous vehicles (CAVs) to exhibit capabilities akin to human cognition, facilitating consistent decision-making across various critical dimensions. The proposed decision-making strategy is validated in challenging environments characterized by high density, randomness, and dynamism on highway roads. We assess the performance of our framework through ablation studies, comparative analyses, and spatiotemporal trajectory evaluations. This study presents a quantitative analysis of decision-making mechanisms mirroring human cognitive functions in the realm of heterogeneous mixed autonomy, promoting the development of multi-dimensional decision-making strategies and a sophisticated distribution of attentional resources.

Figures (9)

• Figure 1: Schematic diagrams of dynamic and multidimensional traffic graphs. Colored and gray circles denote CAVs and HVs, respectively. (a) Spatial dynamics among vehicles alter graph's topological structure from $t$ to $t + 1$, affecting number of nodes and edges, and objectives of edge connections, while categorization of edges remains unchanged; (b) Interactions are distinguished based on vehicle type and spatial dynamics. LC: interaction between CAV and HV in adjacent lane; FL: interaction between CAV and HV in same lane; CC: interaction between two CAVs in adjacent lanes or same lane.
• Figure 2: Proposed multilevel graph reinforcement learning framework. Traffic scenario is represented as multilevel graph. Based on characterization of its multidimensional graphs, parallel asynchronous hierarchical graph reinforcement learning algorithm is proposed for training actions in both lane-changing and following dimensions. Actions are enhanced in exploration efficiency through exponential exploration mechanism, and subject to centralized control system. Hierarchical reward functions computes reward feedback for corresponding dimensions, facilitating policy iteration updates.
• Figure 3: Asynchronous Multidimensional Graphical Markov Decision Process.
• Figure 4: Three-lane highway entry/exit scenario. CAVs enter main lane from originating point, change lanes and speed by judging gap in traffic flow, and exit from destination point.
• Figure 5: Variations in rewards of two decision-making dimensions over 1000 episodes in ablation training experiments.

Theorems & Definitions (5)

• Definition 1: Dynamic Traffic Graph
• Theorem 1: Equilibrium Theorem for Degree Distribution in Weighted Dynamic Graphs
• Definition 2: Multidimensional Traffic Graph
• Theorem 2: Targeted Attention-Driven Entropy Reduction in Sub-Dimensional Graphs
• Definition 3: AMG-MDP