Mitigating Partial Observability in Adaptive Traffic Signal Control with Transformers

TL;DR

This work addresses partial observability in Adaptive Traffic Signal Control (ATSC) by integrating Transformer-based encoders into the multi-agent RL framework eMARLIN to leverage historical observations for improved coordination. It treats the ATSC problem as a $POMDP$ and uses attention over history to overcome sensing limits, highlighting the benefits of Transformer over LSTM-based history handling. The main contributions include the eMARLIN-Transformer architecture, implementation strategies for history integration and masking, and empirical demonstrations of improved coordination and reduced delays on a Toronto/local test-bed. The results indicate that Transformer-based encoders can effectively mitigate PO in urban traffic networks, yielding practical improvements for real-world traffic management.

Abstract

Efficient traffic signal control is essential for managing urban transportation, minimizing congestion, and improving safety and sustainability. Reinforcement Learning (RL) has emerged as a promising approach to enhancing adaptive traffic signal control (ATSC) systems, allowing controllers to learn optimal policies through interaction with the environment. However, challenges arise due to partial observability (PO) in traffic networks, where agents have limited visibility, hindering effectiveness. This paper presents the integration of Transformer-based controllers into ATSC systems to address PO effectively. We propose strategies to enhance training efficiency and effectiveness, demonstrating improved coordination capabilities in real-world scenarios. The results showcase the Transformer-based model's ability to capture significant information from historical observations, leading to better control policies and improved traffic flow. This study highlights the potential of leveraging the advanced Transformer architecture to enhance urban transportation management.

Figures (2)

• Figure 1: The architecture of a Transformer encoder model.
• Figure 2: The architecture of Encoder in eMARLIN-Transformer.