Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Queue Length and Attention Mechanisms for Enhanced Traffic Signal Control Optimization

Liang Zhang, Shubin Xie, Jianming Deng

TL;DR

This work tackles reinforcement learning for traffic signal control (TSC) and demonstrates that queue length is a compact yet powerful state representation. It introduces two methods: Max Queue-Length (M-QL), an optimization-based policy, and AttentionLight, an RL agent that uses self-attention to learn phase correlations without hand-crafted guidance. Across CityFlow simulations and seven real-world datasets, M-QL achieves strong performance and AttentionLight sets a new state-of-the-art, highlighting that state representation can be as critical as network architecture. The findings advocate for queue-length-centric designs to improve robustness, scalability, and deployability of TSC systems.

Abstract

Reinforcement learning (RL) techniques for traffic signal control (TSC) have gained increasing popularity in recent years. However, most existing RL-based TSC methods tend to focus primarily on the RL model structure while neglecting the significance of proper traffic state representation. Furthermore, some RL-based methods heavily rely on expert-designed traffic signal phase competition. In this paper, we present a novel approach to TSC that utilizes queue length as an efficient state representation. We propose two new methods: (1) Max Queue-Length (M-QL), an optimization-based traditional method designed based on the property of queue length; and (2) AttentionLight, an RL model that employs the self-attention mechanism to capture the signal phase correlation without requiring human knowledge of phase relationships. Comprehensive experiments on multiple real-world datasets demonstrate the effectiveness of our approach: (1) the M-QL method outperforms the latest RL-based methods; (2) AttentionLight achieves a new state-of-the-art performance; and (3) our results highlight the significance of proper state representation, which is as crucial as neural network design in TSC methods. Our findings have important implications for advancing the development of more effective and efficient TSC methods. Our code is released on Github (https://github. com/LiangZhang1996/AttentionLight).

Leveraging Queue Length and Attention Mechanisms for Enhanced Traffic Signal Control Optimization

TL;DR

This work tackles reinforcement learning for traffic signal control (TSC) and demonstrates that queue length is a compact yet powerful state representation. It introduces two methods: Max Queue-Length (M-QL), an optimization-based policy, and AttentionLight, an RL agent that uses self-attention to learn phase correlations without hand-crafted guidance. Across CityFlow simulations and seven real-world datasets, M-QL achieves strong performance and AttentionLight sets a new state-of-the-art, highlighting that state representation can be as critical as network architecture. The findings advocate for queue-length-centric designs to improve robustness, scalability, and deployability of TSC systems.

Abstract

Reinforcement learning (RL) techniques for traffic signal control (TSC) have gained increasing popularity in recent years. However, most existing RL-based TSC methods tend to focus primarily on the RL model structure while neglecting the significance of proper traffic state representation. Furthermore, some RL-based methods heavily rely on expert-designed traffic signal phase competition. In this paper, we present a novel approach to TSC that utilizes queue length as an efficient state representation. We propose two new methods: (1) Max Queue-Length (M-QL), an optimization-based traditional method designed based on the property of queue length; and (2) AttentionLight, an RL model that employs the self-attention mechanism to capture the signal phase correlation without requiring human knowledge of phase relationships. Comprehensive experiments on multiple real-world datasets demonstrate the effectiveness of our approach: (1) the M-QL method outperforms the latest RL-based methods; (2) AttentionLight achieves a new state-of-the-art performance; and (3) our results highlight the significance of proper state representation, which is as crucial as neural network design in TSC methods. Our findings have important implications for advancing the development of more effective and efficient TSC methods. Our code is released on Github (https://github. com/LiangZhang1996/AttentionLight).
Paper Structure (25 sections, 1 theorem, 9 equations, 5 figures, 1 table)

This paper contains 25 sections, 1 theorem, 9 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Traditional Methods
  4. RL-Based Methods
  5. Preliminary
  6. Method
  7. Introduce Queue Length for TSC Methods
  8. Property of Queue Length
  9. Max-QueueLength Control
  10. M-QL can stabilize the network
  11. Comparison of M-QL and MP
  12. AttentionLight Agent Design
  13. Advanced RL framework
  14. Network Design of AttentionLight
  15. AttentionLight and FRAP
  16. ...and 10 more sections

Key Result

theorem thmcountertheorem

The M-QL control $u^{*}$ is stabilizing whenever the average demand is admissibleAn admissible demand means the traffic demand can be accommodated by traffic signal control policies, not including situations like long-lasting over-saturated traffic that requires perimeter control to stop traffic get

Figures (5)

  • Figure 1: The illustration of an intersection. In case (a), phase #2 is activated.
  • Figure 2: Model performance comparison.
  • Figure 3: Model performance under different rewards w.r.t average travel time, the smaller the better.
  • Figure 4: Model performance under different action duration.
  • Figure 5: The average travel time of transfer divided by the average travel time of direct training. The error bars represent the 95% confidence interval for the average travel time ratio.

Theorems & Definitions (2)

  • theorem thmcountertheorem
  • proof