Deep reinforcement learning-based longitudinal control strategy for automated vehicles at signalised intersections

TL;DR

This work develops a deep reinforcement learning framework for longitudinal control of autonomous vehicles at signalised intersections, integrating a comprehensive reward function that captures safety (TTC), efficiency (distance headway), amber-light decision-making, and asymmetric acceleration, trained on a blend of real leader-vehicle trajectories and Ornstein–Uhlenbeck simulated trajectories. By implementing both DDPG and SAC, the study demonstrates robust performance in safety, efficiency, and comfort, with DDPG often delivering smoother acceleration profiles. The approach addresses key gaps in SI DRL research, including amber-phase decisions, stop-and-go dynamics, and the use of mixed data for training, and is validated via ECDF comparisons and safety-critical scenario tests. The findings suggest that DRL-based longitudinal control can enhance traffic safety and efficiency at intersections, offering a pathway toward scalable, robust autonomous driving in urban environments, with potential extensions to eco-driving and larger real-world datasets.

Abstract

Developing an autonomous vehicle control strategy for signalised intersections (SI) is one of the challenging tasks due to its inherently complex decision-making process. This study proposes a Deep Reinforcement Learning (DRL) based longitudinal vehicle control strategy at SI. A comprehensive reward function has been formulated with a particular focus on (i) distance headway-based efficiency reward, (ii) decision-making criteria during amber light, and (iii) asymmetric acceleration/ deceleration response, along with the traditional safety and comfort criteria. This reward function has been incorporated with two popular DRL algorithms, Deep Deterministic Policy Gradient (DDPG) and Soft-Actor Critic (SAC), which can handle the continuous action space of acceleration/deceleration. The proposed models have been trained on the combination of real-world leader vehicle (LV) trajectories and simulated trajectories generated using the Ornstein-Uhlenbeck (OU) process. The overall performance of the proposed models has been tested using Cumulative Distribution Function (CDF) plots and compared with the real-world trajectory data. The results show that the RL models successfully maintain lower distance headway (i.e., higher efficiency) and jerk compared to human-driven vehicles without compromising safety. Further, to assess the robustness of the proposed models, we evaluated the model performance on diverse safety-critical scenarios, in terms of car-following and traffic signal compliance. Both DDPG and SAC models successfully handled the critical scenarios, while the DDPG model showed smoother action profiles compared to the SAC model. Overall, the results confirm that DRL-based longitudinal vehicle control strategy at SI can help to improve traffic safety, efficiency, and comfort.

Figures (12)

• Figure 1: Overview of state representation used in proposed RL framework
• Figure 2: Illustration of vehicle behaviour and penalty conditions when approaching to traffic light
• Figure 3: Probability density function (pdf) of a log-normal distribution
• Figure 4: Reward function for distance headway using a lognormal distribution
• Figure 5: Visualisation of reward function for (a) action and (b) jerk in comfortable driving