HALO: Hierarchical Reinforcement Learning for Large-Scale Adaptive Traffic Signal Control

Abstract

Adaptive traffic signal control (ATSC) is essential for mitigating urban congestion in modern smart cities, where traffic infrastructure is evolving into interconnected Web-of-Things (WoT) environments with thousands of sensing-and-control nodes. However, existing methods face a critical scalability-coordination tradeoff: centralized approaches optimize global objectives but become computationally intractable at city scale, while decentralized multi-agent methods scale efficiently yet lack network-level coherence, resulting in suboptimal performance. In this paper, we present HALO, a hierarchical reinforcement learning framework that addresses this tradeoff for large-scale ATSC. HALO decouples decision-making into two levels: a high-level global guidance policy employs Transformer-LSTM encoders to model spatio-temporal dependencies across the entire network and broadcast compact guidance signals, while low-level local intersection policies execute decentralized control conditioned on both local observations and global context. To ensure better alignment of global-local objectives, we introduce an adversarial goal-setting mechanism where the global policy proposes challenging-yet-feasible network-level targets that local policies are trained to surpass, fostering robust coordination. We evaluate HALO extensively on multiple standard benchmarks, and a newly constructed large-scale Manhattan-like network with 2,668 intersections under real-world traffic patterns, including peak transitions, adverse weather and holiday surges. Results demonstrate HALO shows competitive performance and becomes increasingly dominant as network complexity grows across small-scale benchmarks, while delivering the strongest performance in all large-scale regimes, offering up to 6.8% lower average travel time and 5.0% lower average delay than the best state-of-the-art.

Figures (8)

• Figure 1: (a) Traditional centralized vs. decentralized (multi-agnet) reinforcement learning (RL) paradigms for ATSC. (b) The proposed HALO framework, which uses a global policy to provide guidance to decentralized local policies.
• Figure 2: Four-leg signalized intersection with eight non-conflicting phases (A–H).
• Figure 3: An overview of the two-level HALO pipeline. The global policy (yellow) compresses regional states with a Transformer and LSTMs to produce a broadcast guidance $F_g$ and a measurable sub-goal $G^t$; the local intersection policy (blue) forms per-junction observations from the target node and its nearest neighbors, applies MLP + direction-aware graph attention, appends $F_g$ for Actor–Critic control; adversarial losses $\mathcal{L}_{\text{Global}}, \mathcal{L}_{\text{AC}}, \mathcal{L}_{\text{Goal}}$ couple the two levels and drive the next-state feedback.
• Figure 4: Relative gains (%) of HALO vs. the strongest baseline ordered by road network sizes (left to right, small to large). Color encodes improvement (darker means better).
• Figure 5: Episode–reward curves of HALO variants (a) with different components removed; and (b) with and without pretraining of the global guidance policy.