Extensive Exploration in Complex Traffic Scenarios using Hierarchical Reinforcement Learning

TL;DR

The paper tackles autonomous highway driving in complex traffic by introducing a hierarchical DRL framework that splits decision-making into long-horizon high-level planning and precise low-level control. A two-step training procedure first optimizes the high-level policy and then the low-level controller, enabling better exploration and long-term rewards. Key contributions include a novel hierarchical DRL architecture, a speed-biased reward design, and a trap-scenario evaluation demonstrating improved long-term performance over single-level DRL. This approach enhances robustness and generalization in realistic traffic conditions, informing safer and scalable autonomous driving strategies.

Abstract

Developing an automated driving system capable of navigating complex traffic environments remains a formidable challenge. Unlike rule-based or supervised learning-based methods, Deep Reinforcement Learning (DRL) based controllers eliminate the need for domain-specific knowledge and datasets, thus providing adaptability to various scenarios. Nonetheless, a common limitation of existing studies on DRL-based controllers is their focus on driving scenarios with simple traffic patterns, which hinders their capability to effectively handle complex driving environments with delayed, long-term rewards, thus compromising the generalizability of their findings. In response to these limitations, our research introduces a pioneering hierarchical framework that efficiently decomposes intricate decision-making problems into manageable and interpretable subtasks. We adopt a two step training process that trains the high-level controller and low-level controller separately. The high-level controller exhibits an enhanced exploration potential with long-term delayed rewards, and the low-level controller provides longitudinal and lateral control ability using short-term instantaneous rewards. Through simulation experiments, we demonstrate the superiority of our hierarchical controller in managing complex highway driving situations.

Figures (8)

• Figure 1: Experiment Setting to Evaluate Exploration Ability: In this setup, the DRL agent initially encounters a group of two slow-moving traffic vehicles. This scenario tests the agent's ability to navigate through this 'trap'. Upon successfully maneuvering out of this situation, the agent is then introduced to normal traffic conditions, further assessing its adaptability and exploration capabilities.
• Figure 2: Hierarchical DRL framework for highway driving.
• Figure 3: Two-Step Training Process for High-Level and Low-Level Frameworks: Initially, the high-level controller is trained using a model-based motion planner and a critic function. Subsequently, this trained high-level controller is employed to facilitate the training of the low-level controller.
• Figure 4: The trap vehicles are initialized with a longitudinal distance with respect to the ego vehicle.
• Figure 5: Single-level, hierarchical DRL, and h-DQN controller performance during training episodes. (a) Average success rate in evading low-speed traffic: the hierarchical DRL controller exhibits a higher likelihood of successfully navigating out of situations encumbered by low-speed traffic. (b) Average reward: The hierarchical DRL demonstrates superior performance, yielding higher average rewards per episode. (c) Average speed: The hierarchical DRL controller consistently achieves a higher average speed.