Enhancing Cyber Resilience of Networked Microgrids using Vertical Federated Reinforcement Learning

TL;DR

The paper addresses cyber resiliency for networked microgrids under adversarial perturbations to primary-control references. It introduces a vertically federated reinforcement learning approach (FedSAC) to train multi-agent controllers while preserving data privacy, leveraging a GridLAB-D/HELICS co-simulation platform compatible with OpenAI Gym. Key contributions include the FedSAC algorithm, a resilient co-simulation module, and validation on a three-microgrid IEEE-123-bus benchmark demonstrating improved training stability and recovery performance versus decentralized baselines. The work enables privacy-preserving, scalable cyber-resilience for coupled microgrid networks with realistic dynamics.

Abstract

This paper presents a novel federated reinforcement learning (Fed-RL) methodology to enhance the cyber resiliency of networked microgrids. We formulate a resilient reinforcement learning (RL) training setup which (a) generates episodic trajectories injecting adversarial actions at primary control reference signals of the grid forming (GFM) inverters and (b) trains the RL agents (or controllers) to alleviate the impact of the injected adversaries. To circumvent data-sharing issues and concerns for proprietary privacy in multi-party-owned networked grids, we bring in the aspects of federated machine learning and propose a novel Fed-RL algorithm to train the RL agents. To this end, the conventional horizontal Fed-RL approaches using decoupled independent environments fail to capture the coupled dynamics in a networked microgrid, which leads us to propose a multi-agent vertically federated variation of actor-critic algorithms, namely federated soft actor-critic (FedSAC) algorithm. We created a customized simulation setup encapsulating microgrid dynamics in the GridLAB-D/HELICS co-simulation platform compatible with the OpenAI Gym interface for training RL agents. Finally, the proposed methodology is validated with numerical examples of modified IEEE 123-bus benchmark test systems consisting of three coupled microgrids.

Figures (4)

• Figure 1: Resilient RL Co-simulation Platform for Microgrids
• Figure 2: Fed-RL framework for Networked Microgrid
• Figure 3: (a) Agent-wise reward plot for Federated SAC training for 3 different seeds, (b) Comparison of Federated SAC and Multi-agent Decentralized SAC.
• Figure 4: (a) Histogram of Test Rewards, (b) Voltage plot for Bus-1.