Multi-hop Upstream Anticipatory Traffic Signal Control with Deep Reinforcement Learning

TL;DR

Simulations on synthetic and realistic scenarios demonstrate controllers utilizing multi-hop upstream pressure significantly reduce overall network delay by prioritizing traffic movements based on a broader understanding of upstream congestion.

Abstract

Coordination in traffic signal control is crucial for managing congestion in urban networks. Existing pressure-based control methods focus only on immediate upstream links, leading to suboptimal green time allocation and increased network delays. However, effective signal control inherently requires coordination across a broader spatial scope, as the effect of upstream traffic should influence signal control decisions at downstream intersections, impacting a large area in the traffic network. Although agent communication using neural network-based feature extraction can implicitly enhance spatial awareness, it significantly increases the learning complexity, adding an additional layer of difficulty to the challenging task of control in deep reinforcement learning. To address the issue of learning complexity and myopic traffic pressure definition, our work introduces a novel concept based on Markov chain theory, namely \textit{multi-hop upstream pressure}, which generalizes the conventional pressure to account for traffic conditions beyond the immediate upstream links. This farsighted and compact metric informs the deep reinforcement learning agent to preemptively clear the multi-hop upstream queues, guiding the agent to optimize signal timings with a broader spatial awareness. Simulations on synthetic and realistic (Toronto) scenarios demonstrate controllers utilizing multi-hop upstream pressure significantly reduce overall network delay by prioritizing traffic movements based on a broader understanding of upstream congestion.

Figures (11)

• Figure 1: A motivating example to demonstrate the issue of existing myopic pressure definition and the need for farsighted multi-hop upstream pressure definition.
• Figure 2: An example of graph representation for a toy traffic network. (a) A traffic network with 8 traffic links indexed from 0 to 7. (b) The weights shown on the edges are the fabricated turning ratios. The vertex $\Omega$ is the supersink, and the edges in dashed lines represent graph $G^e$ being extended from graph $G$.
• Figure 3: Tested synthetic arterial networks. Link channelization and phasing scheme are described in Section \ref{['sec:exp-setup']}-\ref{['sec:test-scenario']}.
• Figure 4: The Toronto network testbed. Four consecutive intersections on the Sheppard Avenue corridor are controlled by our methods as they encounter large flows. The other eight intersections not experiencing heavy congestion are less critical, therefore are controlled by the city plan.
• Figure 5: TTS vs Episode Reward.

Theorems & Definitions (2)

• Definition 1: Graph representation
• Definition 2: Phase Pressure