AMM: Adaptive Modularized Reinforcement Model for Multi-city Traffic Signal Control

TL;DR

The paper tackles multi-city traffic signal control (TSC) under domain shift by proposing Adaptive Modularized Model (AMM), which decouples input observations from internal dynamics through three modules: representation transition, internal dynamics, and value evaluation. Multi-city data are leveraged via Model-Agnostic Meta-Learning (MAML) to train the dynamics module, enabling rapid adaptation to a new city with limited interactions. Key contributions include modularization to handle varying observations, multi-city experience aggregation with MAML, and extensive experiments on CityFlow datasets demonstrating state-of-the-art performance with minimal target-domain data. The approach offers practical advantages for real-world deployment by reducing data collection costs and enhancing generalization across diverse urban environments.

Abstract

Traffic signal control (TSC) is an important and widely studied direction. Recently, reinforcement learning (RL) methods have been used to solve TSC problems and achieve superior performance over conventional TSC methods. However, applying RL methods to the real world is challenging due to the huge cost of experiments in real-world traffic environments. One possible solution is TSC domain adaptation, which adapts trained models to target environments and reduces the number of interactions and the training cost. However, existing TSC domain adaptation methods still face two major issues: the lack of consideration for differences across cities and the low utilization of multi-city data. To solve aforementioned issues, we propose an approach named Adaptive Modularized Model (AMM). By modularizing TSC problems and network models, we overcome the challenge of possible changes in environmental observations. We also aggregate multi-city experience through meta-learning. We conduct extensive experiments on different cities and show that AMM can achieve excellent performance with limited interactions in target environments and outperform existing methods. We also demonstrate the feasibility and generalizability of our method.

Figures (7)

• Figure 1: The provided observations are different across different traffic environments.
• Figure 2: The examples of observations, actions, states.
• Figure 3: The overview of the method. First, learn dynamics and value-evaluating modules from multi-city data. Then adapt the trained model to the target city by fine-tuning. Finally, use the adaptive model to control the traffic light in the target city. The method is able to achieve SOTA performance with a small amount of interaction with the target city.
• Figure 4: The modularized network model. The model includes three modules: representation transition module, internal dynamics module, and value-evaluating module.
• Figure 5: The comparison of average travel time between methods on different volumes of provided interactive data.