Integrated Freeway Traffic Control Using Q-Learning with Adjacent Arterial Traffic Considerations
Tianchen Yuan, Petros A. Ioannou
TL;DR
The paper tackles integrated control of freeway and adjacent arterial traffic using a model-free Q-learning framework to coordinate VSL, LC, and RM while using arterial signal timing and demand as part of the state. The proposed freeway traffic control agent is trained offline on a small network and then deployed online in a larger connected network for continuous learning, with arterial control modeled via a traffic-responsive cycle-length approach. Key contributions include a Q-learning-based FTC that reduces freeway travel time and stops in congested or incident scenarios and unexpectedly also trims arterial queues through improved off-ramp processing; the work sets the stage for a fully integrated control of arterial signals within the QL framework. The approach demonstrates practical impact by leveraging real-world demands and microscopic simulation, and points to future work on jointly optimizing arterial signal plans within the learning framework.
Abstract
Numerous studies have shown the effectiveness of intelligent transportation system techniques such as variable speed limit (VSL), lane change (LC) control, and ramp metering (RM) in freeway traffic flow control. The integration of these techniques has the potential to further enhance the traffic operation efficiency of both freeway and adjacent arterial networks. In this regard, we propose a freeway traffic control (FTC) strategy that coordinates VSL, LC, RM actions using a Q-learning (QL) framework which takes into account arterial traffic characteristics. The signal timing and demands of adjacent arterial intersections are incorporated as state variables of the FTC agent. The FTC agent is initially trained offline using a single-section road network, and subsequently deployed online in a connected freeway and arterial simulation network for continuous learning. The arterial network is assumed to be regulated by a traffic-responsive signal control strategy based on a cycle length model. Microscopic simulations demonstrate that the fully-trained FTC agent provides significant reductions in freeway travel time and the number of stops in scenarios with traffic congestion. It clearly outperforms an uncoordinated FTC and a decentralized feedback control strategy. Even though the FTC agent does not control the arterial traffic signals, it leads to shorter average queue lengths at arterial intersections by taking into account the arterial traffic conditions in controlling freeway traffic. These results motivate a future research where the QL framework will also include the control of arterial traffic signals.