Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep Reinforcement Learning for Traffic Light Control in Intelligent Transportation Systems

Ming Zhu, Xiao-Yang Liu, Sem Borst, Anwar Walid

TL;DR

This work tackles real-time traffic-light control in ITS, addressing scalability challenges of traditional methods. A dual-DRL approach is proposed: DQN is applied to a single intersection to obtain optimal-like threshold policies, while DDPG is scaled to a grid network where an emergent greenwave policy is observed and proven optimal under a symmetric fluid-traffic model. The study provides both theoretical guarantees and numerical evidence that DRL can yield scalable, high-level traffic-control strategies, including the synchronized greens across intersections in a grid. The findings highlight the potential of DRL for large-scale traffic management and offer a foundation for future extensions to stochastic arrivals, distributed coordination, and edge-based implementations.

Abstract

Smart traffic lights in intelligent transportation systems (ITSs) are envisioned to greatly increase traffic efficiency and reduce congestion. Deep reinforcement learning (DRL) is a promising approach to adaptively control traffic lights based on the real-time traffic situation in a road network. However, conventional methods may suffer from poor scalability. In this paper, we investigate deep reinforcement learning to control traffic lights, and both theoretical analysis and numerical experiments show that the intelligent behavior ``greenwave" (i.e., a vehicle will see a progressive cascade of green lights, and not have to brake at any intersection) emerges naturally a grid road network, which is proved to be the optimal policy in an avenue with multiple cross streets. As a first step, we use two DRL algorithms for the traffic light control problems in two scenarios. In a single road intersection, we verify that the deep Q-network (DQN) algorithm delivers a thresholding policy; and in a grid road network, we adopt the deep deterministic policy gradient (DDPG) algorithm. Secondly, numerical experiments show that the DQN algorithm delivers the optimal control, and the DDPG algorithm with passive observations has the capability to produce on its own a high-level intelligent behavior in a grid road network, namely, the ``greenwave" policy emerges. We also verify the ``greenwave" patterns in a $5 \times 10$ grid road network. Thirdly, the ``greenwave" patterns demonstrate that DRL algorithms produce favorable solutions since the ``greenwave" policy shown in experiment results is proved to be optimal in a specified traffic model (an avenue with multiple cross streets). The delivered policies both in a single road intersection and a grid road network demonstrate the scalability of DRL algorithms.

Deep Reinforcement Learning for Traffic Light Control in Intelligent Transportation Systems

TL;DR

This work tackles real-time traffic-light control in ITS, addressing scalability challenges of traditional methods. A dual-DRL approach is proposed: DQN is applied to a single intersection to obtain optimal-like threshold policies, while DDPG is scaled to a grid network where an emergent greenwave policy is observed and proven optimal under a symmetric fluid-traffic model. The study provides both theoretical guarantees and numerical evidence that DRL can yield scalable, high-level traffic-control strategies, including the synchronized greens across intersections in a grid. The findings highlight the potential of DRL for large-scale traffic management and offer a foundation for future extensions to stochastic arrivals, distributed coordination, and edge-based implementations.

Abstract

Smart traffic lights in intelligent transportation systems (ITSs) are envisioned to greatly increase traffic efficiency and reduce congestion. Deep reinforcement learning (DRL) is a promising approach to adaptively control traffic lights based on the real-time traffic situation in a road network. However, conventional methods may suffer from poor scalability. In this paper, we investigate deep reinforcement learning to control traffic lights, and both theoretical analysis and numerical experiments show that the intelligent behavior ``greenwave" (i.e., a vehicle will see a progressive cascade of green lights, and not have to brake at any intersection) emerges naturally a grid road network, which is proved to be the optimal policy in an avenue with multiple cross streets. As a first step, we use two DRL algorithms for the traffic light control problems in two scenarios. In a single road intersection, we verify that the deep Q-network (DQN) algorithm delivers a thresholding policy; and in a grid road network, we adopt the deep deterministic policy gradient (DDPG) algorithm. Secondly, numerical experiments show that the DQN algorithm delivers the optimal control, and the DDPG algorithm with passive observations has the capability to produce on its own a high-level intelligent behavior in a grid road network, namely, the ``greenwave" policy emerges. We also verify the ``greenwave" patterns in a grid road network. Thirdly, the ``greenwave" patterns demonstrate that DRL algorithms produce favorable solutions since the ``greenwave" policy shown in experiment results is proved to be optimal in a specified traffic model (an avenue with multiple cross streets). The delivered policies both in a single road intersection and a grid road network demonstrate the scalability of DRL algorithms.
Paper Structure (18 sections, 4 theorems, 71 equations, 6 figures, 3 tables)

This paper contains 18 sections, 4 theorems, 71 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Conventional Methods
  4. Deep Reinforcement Learning Methods
  5. Model Description and Problem Formulation
  6. Model for Single Traffic Light
  7. Model for Multiple Traffic Lights
  8. Objective Function for Optimization
  9. Deep Reinforcement Learning Algorithms
  10. Reinforcement Learning and Q-Learning
  11. DQN for Single Road Intersection
  12. Scalable DDPG for Grid Road Network
  13. Optimality of the "Greenwave" Policy
  14. Performance Evaluation
  15. Experiment Settings
  16. ...and 3 more sections

Key Result

Lemma 1

For all $n \in \{ 1, ..., N \}$, The above inequations are strict unless $G_n = p_n (Y_n + O_n)$ and $R_n = q_n (Y_n + O_n)$, respectively.

Figures (6)

  • Figure 1: An intersection with two traffic flows (left), and the corresponding state-transition diagram (right).
  • Figure 2: Illustration of a road network with grid topology.
  • Figure 3: Performance of DQN control policy (left), fixed-cycle policy (middle), and the optimal policy (right) for single intersection. Tests of 300 steps in simulation are shown with average reward and number of waiting cars. Better policies are supposed to reduce the numbers of waiting cars. The DQN algorithm matches the optimal policy.
  • Figure 4: Comparison for a single intersection with different car arrival/passing ratio.
  • Figure 5: Stabilization of a single intersection with large initial queue length.
  • ...and 1 more figures

Theorems & Definitions (9)

  • Lemma 1
  • proof
  • Remark 1
  • Lemma 2
  • proof
  • Proposition 1
  • proof
  • Proposition 2
  • proof