Optimizing Electric Bus Charging Scheduling with Uncertainties Using Hierarchical Deep Reinforcement Learning

TL;DR

This paper tackles electric bus (EB) charging scheduling under uncertainties in travel time, energy use, and real-time electricity prices. It introduces a hierarchical deep reinforcement learning (HDRL) framework that reformulates the problem into two augmented MDPs and presents the DAC-MAPPO-E algorithm, combining a centralized high-level policy with decentralized low-level MAPPO agents under CTDE. Key contributions include an attention-enhanced high-level actor, a scalable two-level architecture, and a complexity analysis showing substantial improvements over naive DAC-MAPPO; experiments on real-world price data demonstrate performance near a full MILP oracle with superior scalability. The work shows HDRL can deliver efficient, scalable, and adaptive EB charging schedules that can adapt to dynamic grid conditions and multi-time-scale decision requirements, enabling practical deployment in large fleets.

Abstract

The growing adoption of Electric Buses (EBs) represents a significant step toward sustainable development. By utilizing Internet of Things (IoT) systems, charging stations can autonomously determine charging schedules based on real-time data. However, optimizing EB charging schedules remains a critical challenge due to uncertainties in travel time, energy consumption, and fluctuating electricity prices. Moreover, to address real-world complexities, charging policies must make decisions efficiently across multiple time scales and remain scalable for large EB fleets. In this paper, we propose a Hierarchical Deep Reinforcement Learning (HDRL) approach that reformulates the original Markov Decision Process (MDP) into two augmented MDPs. To solve these MDPs and enable multi-timescale decision-making, we introduce a novel HDRL algorithm, namely Double Actor-Critic Multi-Agent Proximal Policy Optimization Enhancement (DAC-MAPPO-E). Scalability challenges of the Double Actor-Critic (DAC) algorithm for large-scale EB fleets are addressed through enhancements at both decision levels. At the high level, we redesign the decentralized actor network and integrate an attention mechanism to extract relevant global state information for each EB, decreasing the size of neural networks. At the low level, the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm is incorporated into the DAC framework, enabling decentralized and coordinated charging power decisions, reducing computational complexity and enhancing convergence speed. Extensive experiments with real-world data demonstrate the superior performance and scalability of DAC-MAPPO-E in optimizing EB fleet charging schedules.

Figures (5)

• Figure 1: The schematic diagram of the system model.
• Figure 2: The new decentralized high-level actor network is designed by decoupling the high-level action space, with an architecture comprising $M$ agent networks and a pair of mapping networks. Each agent network is associated with an EB. The mapping networks are utilized to derive the policy over options $\mu(o_t|S_t)$.
• Figure 3: The attention layer is utilized to compress the global state $S_t$ to $(S_{i,t},S_{i,t}^{\rm att})$.
• Figure 4: The performance curves of DRL-based algorithms. The shaded areas represent the standard errors across three runs.
• Figure 5: The detailed charging schedules of DAC-MAPPO-E, MILP-S, and PPO-MILP for Scenario 1.

Theorems & Definitions (2)

• Definition 1: High-Level MDP
• Definition 2: Low-Level MDP