Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Cyber-Resilience in Integrated Energy System Scheduling with Demand Response Using Deep Reinforcement Learning

Yang Li, Wenjie Ma, Yuanzheng Li, Sen Li, Zhe Chen, Mohammad Shahidehpor

TL;DR

The state-adversarial soft actor-critic (SA-SAC) algorithm is proposed to mitigate the impact of cyber-attacks on the scheduling strategy, integrating adversarial training into the learning process to against cyber-attacks.

Abstract

Optimally scheduling multi-energy flow is an effective method to utilize renewable energy sources (RES) and improve the stability and economy of integrated energy systems (IES). However, the stable demand-supply of IES faces challenges from uncertainties that arise from RES and loads, as well as the increasing impact of cyber-attacks with advanced information and communication technologies adoption. To address these challenges, this paper proposes an innovative model-free resilience scheduling method based on state-adversarial deep reinforcement learning (DRL) for integrated demand response (IDR)-enabled IES. The proposed method designs an IDR program to explore the interaction ability of electricity-gas-heat flexible loads. Additionally, the state-adversarial Markov decision process (SA-MDP) model characterizes the energy scheduling problem of IES under cyber-attack, incorporating cyber-attacks as adversaries directly into the scheduling process. The state-adversarial soft actor-critic (SA-SAC) algorithm is proposed to mitigate the impact of cyber-attacks on the scheduling strategy, integrating adversarial training into the learning process to against cyber-attacks. Simulation results demonstrate that our method is capable of adequately addressing the uncertainties resulting from RES and loads, mitigating the impact of cyber-attacks on the scheduling strategy, and ensuring a stable demand supply for various energy sources. Moreover, the proposed method demonstrates resilience against cyber-attacks. Compared to the original soft actor-critic (SAC) algorithm, it achieves a 10% improvement in economic performance under cyber-attack scenarios.

Enhancing Cyber-Resilience in Integrated Energy System Scheduling with Demand Response Using Deep Reinforcement Learning

TL;DR

The state-adversarial soft actor-critic (SA-SAC) algorithm is proposed to mitigate the impact of cyber-attacks on the scheduling strategy, integrating adversarial training into the learning process to against cyber-attacks.

Abstract

Optimally scheduling multi-energy flow is an effective method to utilize renewable energy sources (RES) and improve the stability and economy of integrated energy systems (IES). However, the stable demand-supply of IES faces challenges from uncertainties that arise from RES and loads, as well as the increasing impact of cyber-attacks with advanced information and communication technologies adoption. To address these challenges, this paper proposes an innovative model-free resilience scheduling method based on state-adversarial deep reinforcement learning (DRL) for integrated demand response (IDR)-enabled IES. The proposed method designs an IDR program to explore the interaction ability of electricity-gas-heat flexible loads. Additionally, the state-adversarial Markov decision process (SA-MDP) model characterizes the energy scheduling problem of IES under cyber-attack, incorporating cyber-attacks as adversaries directly into the scheduling process. The state-adversarial soft actor-critic (SA-SAC) algorithm is proposed to mitigate the impact of cyber-attacks on the scheduling strategy, integrating adversarial training into the learning process to against cyber-attacks. Simulation results demonstrate that our method is capable of adequately addressing the uncertainties resulting from RES and loads, mitigating the impact of cyber-attacks on the scheduling strategy, and ensuring a stable demand supply for various energy sources. Moreover, the proposed method demonstrates resilience against cyber-attacks. Compared to the original soft actor-critic (SAC) algorithm, it achieves a 10% improvement in economic performance under cyber-attack scenarios.
Paper Structure (41 sections, 1 theorem, 61 equations, 15 figures, 3 tables, 1 algorithm)

This paper contains 41 sections, 1 theorem, 61 equations, 15 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Modeling of IES and LR Attacks
  3. Structure Modeling of IES
  4. Modeling of Load
  5. Electric Load Model
  6. Gas Load Model
  7. Heat Load Model
  8. Energy Storage Device Model
  9. Electric Boiler Model
  10. Power to Gas Model
  11. Micro-Gas Turbine Model
  12. Modeling of LR Attack
  13. Bad Data Detection in IES State Estimation
  14. HLR Attack Modeling
  15. Formulation of IES scheduling model
  16. ...and 26 more sections

Key Result

Theorem 1

For a non-adversarial MDP with a policy $\pi$, and an optimal adversary $v$ in SA-MDP, the following holds for all states $s\in S$: where ${{\text{D}}_{TV}}(\pi (\cdot \mid s),\pi (\cdot \mid \hat{s}))$ denotes the total variation distance between $\pi (\cdot \mid s)$and $\pi (\cdot \mid \hat{s})$; $V^\pi(s)$ and $\tilde{V}_v^\pi(s)$ represent the system performance under conditions without and w

Figures (15)

  • Figure 1: Schematic diagram of the IES.
  • Figure 2: State-Adversarial Markov decision process.
  • Figure 3: The IES scheduling framework based on the proposed SA-SAC.
  • Figure 4: Robustness comparison of the SA-SAC and SAC to cyber-attack.
  • Figure 5: Daily temperature curve and energy price.
  • ...and 10 more figures

Theorems & Definitions (1)

  • Theorem 1