Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DRNet: A Decision-Making Method for Autonomous Lane Changingwith Deep Reinforcement Learning

Kunpeng Xu, Lifei Chen, Shengrui Wang

TL;DR

DRNet addresses autonomous vehicle lane-changing by formulating it as a tactical decision-making problem and developing a DRL-based framework that integrates safety into learning. It uses a CNN-based feature extractor and CNN+SVM for driving-style identification, coupled with a prioritized replay buffer and an Action-Subspace mechanism to constrain actions to safe options, improving data efficiency and safety. The state is represented as a three-layer occupancy grid, and rewards combine collision penalties and driving-style distances to encourage safe, efficient behavior; DRNet outperforms DDQN and rule-based baselines in simulation and achieves strong real-world prediction performance on NGSIM data. The approach yields faster learning, higher decision-making efficiency, and robust safety guarantees, with practical implications for scalable, safe autonomous driving in multi-lane traffic.

Abstract

Machine learning techniques have outperformed numerous rule-based methods for decision-making in autonomous vehicles. Despite recent efforts, lane changing remains a major challenge, due to the complex driving scenarios and changeable social behaviors of surrounding vehicles. To help improve the state of the art, we propose to leveraging the emerging \underline{D}eep \underline{R}einforcement learning (DRL) approach for la\underline{NE} changing at the \underline{T}actical level. To this end, we present "DRNet", a novel and highly efficient DRL-based framework that enables a DRL agent to learn to drive by executing reasonable lane changing on simulated highways with an arbitrary number of lanes, and considering driving style of surrounding vehicles to make better decisions. Furthermore, to achieve a safe policy for decision-making, DRNet incorporates ideas from safety verification, the most important component of autonomous driving, to ensure that only safe actions are chosen at any time. The setting of our state representation and reward function enables the trained agent to take appropriate actions in a real-world-like simulator. Our DRL agent has the ability to learn the desired task without causing collisions and outperforms DDQN and other baseline models.

DRNet: A Decision-Making Method for Autonomous Lane Changingwith Deep Reinforcement Learning

TL;DR

DRNet addresses autonomous vehicle lane-changing by formulating it as a tactical decision-making problem and developing a DRL-based framework that integrates safety into learning. It uses a CNN-based feature extractor and CNN+SVM for driving-style identification, coupled with a prioritized replay buffer and an Action-Subspace mechanism to constrain actions to safe options, improving data efficiency and safety. The state is represented as a three-layer occupancy grid, and rewards combine collision penalties and driving-style distances to encourage safe, efficient behavior; DRNet outperforms DDQN and rule-based baselines in simulation and achieves strong real-world prediction performance on NGSIM data. The approach yields faster learning, higher decision-making efficiency, and robust safety guarantees, with practical implications for scalable, safe autonomous driving in multi-lane traffic.

Abstract

Machine learning techniques have outperformed numerous rule-based methods for decision-making in autonomous vehicles. Despite recent efforts, lane changing remains a major challenge, due to the complex driving scenarios and changeable social behaviors of surrounding vehicles. To help improve the state of the art, we propose to leveraging the emerging \underline{D}eep \underline{R}einforcement learning (DRL) approach for la\underline{NE} changing at the \underline{T}actical level. To this end, we present "DRNet", a novel and highly efficient DRL-based framework that enables a DRL agent to learn to drive by executing reasonable lane changing on simulated highways with an arbitrary number of lanes, and considering driving style of surrounding vehicles to make better decisions. Furthermore, to achieve a safe policy for decision-making, DRNet incorporates ideas from safety verification, the most important component of autonomous driving, to ensure that only safe actions are chosen at any time. The setting of our state representation and reward function enables the trained agent to take appropriate actions in a real-world-like simulator. Our DRL agent has the ability to learn the desired task without causing collisions and outperforms DDQN and other baseline models.
Paper Structure (22 sections, 1 theorem, 15 equations, 9 figures, 6 tables, 1 algorithm)

This paper contains 22 sections, 1 theorem, 15 equations, 9 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Lane-Changing Decision-making
  4. DRL
  5. System Model and Problem Formulation
  6. Proposed Solution: DRNet
  7. Feature Extraction and Driving style Identification
  8. Convolutional Neural Networks (CNNs):
  9. Support Vector Machines (SVMs):
  10. Integration into the DRL Framework
  11. Initialization and Sampling Transitions
  12. Action-Subspace
  13. Algorithm Descriptions: DRNet
  14. Performance Evaluation
  15. Experimental Setting
  16. ...and 7 more sections

Key Result

Proposition 1

The subspace of safe actions $A_{sub}(t)\subseteq A$ is defined as

Figures (9)

  • Figure 1: Lane-Changing Maneuver
  • Figure 2: Framework Diagram for Reinforcement Learning
  • Figure 3: A layer of state representation
  • Figure 4: DRNet Framework
  • Figure 5: Summary plots of learning speed
  • ...and 4 more figures

Theorems & Definitions (2)

  • Proposition 1: Safe Action Subspace
  • proof