Efficient Mitigation of Bus Bunching through Setter-Based Curriculum Learning
Avidan Shah, Danny Tran, Yuhan Tang
TL;DR
The paper tackles the bus-bunching reinforcement learning problem by introducing a Setter-Based Curriculum Learning framework that automatically manufactures training curricula via a neural setter. The setter jointly modulates environment aspects—action space, adversary perturbations, and initial bus bunching—while training with PPO and enhancing robustness through Domain Randomization. Experimental results show the setter-based approach can outperform some hand-crafted curricula and baselines, though it may underperform the no-curriculum setting in certain runs and can exhibit high variance, indicating a need for further tuning. This work contributes a scalable, automated approach to curriculum design in dynamic transportation systems, with implications for broader RL applications where adaptive task difficulty is beneficial.
Abstract
Curriculum learning has been growing in the domain of reinforcement learning as a method of improving training efficiency for various tasks. It involves modifying the difficulty (lessons) of the environment as the agent learns, in order to encourage more optimal agent behavior and higher reward states. However, most curriculum learning methods currently involve discrete transitions of the curriculum or predefined steps by the programmer or using automatic curriculum learning on only a small subset training such as only on an adversary. In this paper, we propose a novel approach to curriculum learning that uses a Setter Model to automatically generate an action space, adversary strength, initialization, and bunching strength. Transportation and traffic optimization is a well known area of study, especially for reinforcement learning based solutions. We specifically look at the bus bunching problem for the context of this study. The main idea of the problem is to minimize the delays caused by inefficient bus timings for passengers arriving and departing from a system of buses. While the heavy exploration in the area makes innovation and improvement with regards to performance marginal, it simultaneously provides an effective baseline for developing new generalized techniques. Our group is particularly interested in examining curriculum learning and its effect on training efficiency and overall performance. We decide to try a lesser known approach to curriculum learning, in which the curriculum is not fixed or discretely thresholded. Our method for automated curriculum learning involves a curriculum that is dynamically chosen and learned by an adversary network made to increase the difficulty of the agent's training, and defined by multiple forms of input. Our results are shown in the following sections of this paper.