Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Electric Bus Charging Schedules Relying on Real Data-Driven Targets Based on Hierarchical Deep Reinforcement Learning

Jiaju Qi, Lei Lei, Thorsteinn Jonsson, Lajos Hanzo

TL;DR

This work tackles the challenge of minimizing EB charging costs under real-time electricity prices and uncertain operation by proposing a Hierarchical DRL framework that decomposes a long-range MDP into a high-level SMDP for data-driven charging targets and low-level MDPs for fine-grained charging control. The HDDQN-HER algorithm learns both levels concurrently, using two DDQNs and hindsight experience replay to address non-stationarity and sample inefficiency. Theoretical results show that the flat policy formed by composing the optimal high- and low-level policies matches the original MDP optimality, while empirical results on real-price data demonstrate superior performance and faster convergence compared with baselines. The approach enables adaptive, cost-efficient charging strategies that can incorporate V2G and real-time pricing, with potential for fleet-wide extensions and safe-RL guarantees in future work.

Abstract

The charging scheduling problem of Electric Buses (EBs) is investigated based on Deep Reinforcement Learning (DRL). A Markov Decision Process (MDP) is conceived, where the time horizon includes multiple charging and operating periods in a day, while each period is further divided into multiple time steps. To overcome the challenge of long-range multi-phase planning with sparse reward, we conceive Hierarchical DRL (HDRL) for decoupling the original MDP into a high-level Semi-MDP (SMDP) and multiple low-level MDPs. The Hierarchical Double Deep Q-Network (HDDQN)-Hindsight Experience Replay (HER) algorithm is proposed for simultaneously solving the decision problems arising at different temporal resolutions. As a result, the high-level agent learns an effective policy for prescribing the charging targets for every charging period, while the low-level agent learns an optimal policy for setting the charging power of every time step within a single charging period, with the aim of minimizing the charging costs while meeting the charging target. It is proved that the flat policy constructed by superimposing the optimal high-level policy and the optimal low-level policy performs as well as the optimal policy of the original MDP. Since jointly learning both levels of policies is challenging due to the non-stationarity of the high-level agent and the sampling inefficiency of the low-level agent, we divide the joint learning process into two phases and exploit our new HER algorithm to manipulate the experience replay buffers for both levels of agents. Numerical experiments are performed with the aid of real-world data to evaluate the performance of the proposed algorithm.

Electric Bus Charging Schedules Relying on Real Data-Driven Targets Based on Hierarchical Deep Reinforcement Learning

TL;DR

This work tackles the challenge of minimizing EB charging costs under real-time electricity prices and uncertain operation by proposing a Hierarchical DRL framework that decomposes a long-range MDP into a high-level SMDP for data-driven charging targets and low-level MDPs for fine-grained charging control. The HDDQN-HER algorithm learns both levels concurrently, using two DDQNs and hindsight experience replay to address non-stationarity and sample inefficiency. Theoretical results show that the flat policy formed by composing the optimal high- and low-level policies matches the original MDP optimality, while empirical results on real-price data demonstrate superior performance and faster convergence compared with baselines. The approach enables adaptive, cost-efficient charging strategies that can incorporate V2G and real-time pricing, with potential for fleet-wide extensions and safe-RL guarantees in future work.

Abstract

The charging scheduling problem of Electric Buses (EBs) is investigated based on Deep Reinforcement Learning (DRL). A Markov Decision Process (MDP) is conceived, where the time horizon includes multiple charging and operating periods in a day, while each period is further divided into multiple time steps. To overcome the challenge of long-range multi-phase planning with sparse reward, we conceive Hierarchical DRL (HDRL) for decoupling the original MDP into a high-level Semi-MDP (SMDP) and multiple low-level MDPs. The Hierarchical Double Deep Q-Network (HDDQN)-Hindsight Experience Replay (HER) algorithm is proposed for simultaneously solving the decision problems arising at different temporal resolutions. As a result, the high-level agent learns an effective policy for prescribing the charging targets for every charging period, while the low-level agent learns an optimal policy for setting the charging power of every time step within a single charging period, with the aim of minimizing the charging costs while meeting the charging target. It is proved that the flat policy constructed by superimposing the optimal high-level policy and the optimal low-level policy performs as well as the optimal policy of the original MDP. Since jointly learning both levels of policies is challenging due to the non-stationarity of the high-level agent and the sampling inefficiency of the low-level agent, we divide the joint learning process into two phases and exploit our new HER algorithm to manipulate the experience replay buffers for both levels of agents. Numerical experiments are performed with the aid of real-world data to evaluate the performance of the proposed algorithm.
Paper Structure (31 sections, 1 theorem, 41 equations, 8 figures, 5 tables, 4 algorithms)

This paper contains 31 sections, 1 theorem, 41 equations, 8 figures, 5 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. State-of-the-art
  3. Research on EB charging scheduling problem
  4. Hierarchical RL (HRL)
  5. System model
  6. MDP Model
  7. State
  8. Action
  9. Transition Probability
  10. Reward Function
  11. Optimization Objective
  12. Options over MDP
  13. Definition of Options
  14. Decoupling the original MDP
  15. The High-Level SMDP
  16. ...and 16 more sections

Key Result

Theorem 1

The flat policy created by superimposing the optimal high-level policy over options $\mu^*$ and the optimal low-level intra-option policy $\pi_\omega^*$ performs as well as the optimal policy of the original MDP $\pi^{*}$, i.e., where $\omega^*=\mu^*(s)$, $\forall s\in\mathcal{S}$ represents the charging targets prescribed by $\mu^*$. Moreover, $\mu^*$ is the optimal high-level policy over option

Figures (8)

  • Figure 1: Comparison of MILP, RO, GA, and DRL in terms of four key criteria: lower computational complexity, data independency, uncertainty handling, and adaptability to dynamic environments.
  • Figure 2: The contributions of this paper in contrast to the literature on HRL.
  • Figure 3: The schematic diagram of the system model for the EB charging problem.
  • Figure 4: The schematic diagram of the original MDP, high-level SMDP, and low-level MDPs.
  • Figure 5: The framework of HDDQN-HER algorithm.
  • ...and 3 more figures

Theorems & Definitions (2)

  • Theorem 1
  • proof