Federated Hierarchical Reinforcement Learning for Adaptive Traffic Signal Control
Yongjie Fu, Lingyun Zhong, Zifan Li, Xuan Di
TL;DR
This work tackles adaptive traffic signal control in large, heterogeneous urban networks by introducing Hierarchical Federated Reinforcement Learning (HFRL). It adds two algorithms, FedFomoLight and FedClusterLight, that enable personalized or cluster-based model aggregation within a federated RL framework, aligning federation with traffic demand and network topology. Empirical results on synthetic grids and real-world NYC-like networks show that HFRL methods reduce travel and waiting times while lowering communication costs compared to FedAvg and decentralized baselines, with grouping patterns that reflect road structure and demand. The approach offers scalable, privacy-preserving distributed control for urban traffic systems and opens avenues for incorporating pedestrians, deeper hierarchical layers, and robustness to disruptions in future work.
Abstract
Multi-agent reinforcement learning (MARL) has shown promise for adaptive traffic signal control (ATSC), enabling multiple intersections to coordinate signal timings in real time. However, in large-scale settings, MARL faces constraints due to extensive data sharing and communication requirements. Federated learning (FL) mitigates these challenges by training shared models without directly exchanging raw data, yet traditional FL methods such as FedAvg struggle with highly heterogeneous intersections. Different intersections exhibit varying traffic patterns, demands, and road structures, so performing FedAvg across all agents is inefficient. To address this gap, we propose Hierarchical Federated Reinforcement Learning (HFRL) for ATSC. HFRL employs clustering-based or optimization-based techniques to dynamically group intersections and perform FedAvg independently within groups of intersections with similar characteristics, enabling more effective coordination and scalability than standard FedAvg. Our experiments on synthetic and real-world traffic networks demonstrate that HFRL not only outperforms both decentralized and standard federated RL approaches but also identifies suitable grouping patterns based on network structure or traffic demand, resulting in a more robust framework for distributed, heterogeneous systems.