Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Multi-Agent Deep Reinforcement Learning Approach via Physics-Informed Reward for Multi-Microgrid Energy Management

Yuanzheng Li, Shangyang He, Yang Li, Yang Shi, Zhigang Zeng

TL;DR

This article proposes a federated MADRL (F-MADRL) algorithm via the physics-informed reward, which ensures the privacy and the security of data and introduces the federated learning (FL) mechanism to train the F-M ADRL algorithm.

Abstract

The utilization of large-scale distributed renewable energy promotes the development of the multi-microgrid (MMG), which raises the need of developing an effective energy management method to minimize economic costs and keep self energy-sufficiency. The multi-agent deep reinforcement learning (MADRL) has been widely used for the energy management problem because of its real-time scheduling ability. However, its training requires massive energy operation data of microgrids (MGs), while gathering these data from different MGs would threaten their privacy and data security. Therefore, this paper tackles this practical yet challenging issue by proposing a federated multi-agent deep reinforcement learning (F-MADRL) algorithm via the physics-informed reward. In this algorithm, the federated learning (FL) mechanism is introduced to train the F-MADRL algorithm thus ensures the privacy and the security of data. In addition, a decentralized MMG model is built, and the energy of each participated MG is managed by an agent, which aims to minimize economic costs and keep self energy-sufficiency according to the physics-informed reward. At first, MGs individually execute the self-training based on local energy operation data to train their local agent models. Then, these local models are periodically uploaded to a server and their parameters are aggregated to build a global agent, which will be broadcasted to MGs and replace their local agents. In this way, the experience of each MG agent can be shared and the energy operation data is not explicitly transmitted, thus protecting the privacy and ensuring data security. Finally, experiments are conducted on Oak Ridge national laboratory distributed energy control communication lab microgrid (ORNL-MG) test system, and the comparisons are carried out to verify the effectiveness of introducing the FL mechanism and the outperformance of our proposed F-MADRL.

Federated Multi-Agent Deep Reinforcement Learning Approach via Physics-Informed Reward for Multi-Microgrid Energy Management

TL;DR

This article proposes a federated MADRL (F-MADRL) algorithm via the physics-informed reward, which ensures the privacy and the security of data and introduces the federated learning (FL) mechanism to train the F-M ADRL algorithm.

Abstract

The utilization of large-scale distributed renewable energy promotes the development of the multi-microgrid (MMG), which raises the need of developing an effective energy management method to minimize economic costs and keep self energy-sufficiency. The multi-agent deep reinforcement learning (MADRL) has been widely used for the energy management problem because of its real-time scheduling ability. However, its training requires massive energy operation data of microgrids (MGs), while gathering these data from different MGs would threaten their privacy and data security. Therefore, this paper tackles this practical yet challenging issue by proposing a federated multi-agent deep reinforcement learning (F-MADRL) algorithm via the physics-informed reward. In this algorithm, the federated learning (FL) mechanism is introduced to train the F-MADRL algorithm thus ensures the privacy and the security of data. In addition, a decentralized MMG model is built, and the energy of each participated MG is managed by an agent, which aims to minimize economic costs and keep self energy-sufficiency according to the physics-informed reward. At first, MGs individually execute the self-training based on local energy operation data to train their local agent models. Then, these local models are periodically uploaded to a server and their parameters are aggregated to build a global agent, which will be broadcasted to MGs and replace their local agents. In this way, the experience of each MG agent can be shared and the energy operation data is not explicitly transmitted, thus protecting the privacy and ensuring data security. Finally, experiments are conducted on Oak Ridge national laboratory distributed energy control communication lab microgrid (ORNL-MG) test system, and the comparisons are carried out to verify the effectiveness of introducing the FL mechanism and the outperformance of our proposed F-MADRL.
Paper Structure (23 sections, 39 equations, 10 figures, 3 tables)

This paper contains 23 sections, 39 equations, 10 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Theoretical Basis of Reinforcement Learning
  3. The Decentralized Multi-Microgrid Energy Management Model
  4. The Isolated Microgrid Energy Management Model
  5. Conventional Generator
  6. Renewable Energy Generator
  7. Battery
  8. Network Power Loss of MG
  9. Isolated MG Energy Management Model via MDP and Physics-Informed Reward
  10. State
  11. Action
  12. Reward
  13. Decentralized Multi-Microgrid Energy Management Model
  14. Federated Multi-Agent Deep Reinforcement Learning Algorithm
  15. The F-MADRL: Server Part
  16. ...and 8 more sections

Figures (10)

  • Figure 1: The structure of (a) an isolated MG and (b) the ORNL-MG ORNL_MG_Fig.
  • Figure 2: The structure of MMG system.
  • Figure 3: The proposed federated multi-agent deep reinforcement learning algorithm.
  • Figure 4: The procedure of the self-training of MG agent.
  • Figure 5: The reward curve of (a) MG1 agent (b) MG2 agent and (c) MG3 agent during the training.
  • ...and 5 more figures