Resilient Control of Networked Microgrids using Vertical Federated Reinforcement Learning: Designs and Real-Time Test-Bed Validations

TL;DR

This work tackles resilience in networked microgrids facing adversarial cyber-events while preserving data privacy. It introduces a vertical Federated Reinforcement Learning approach (FedSAC) to coordinate multiple microgrid controllers without sharing raw data, and couples this with a new ResRLCoSIM co-simulation platform to train and test policies in GridLAB-D/HELICS and OpenAI Gym environments. The authors demonstrate sim-to-real transfer by implementing trained policies in a Hypersim-based real-time hardware-in-the-loop test-bed, bridging the gap between simulation and practice. Key findings show that FedSAC-based policies can mitigate attacks on grid-forming inverters and stabilize voltages under varied scenarios, with successful real-time validation and improved resilience compared to decentralized baselines. The work advances privacy-preserving, data-driven resilience for multi-party owned microgrids and provides a practical pathway for deploying learned controls in real-world hardware-in-the-loop settings.

Abstract

Improving system-level resiliency of networked microgrids is an important aspect with increased population of inverter-based resources (IBRs). This paper (1) presents resilient control design in presence of adversarial cyber-events, and proposes a novel federated reinforcement learning (Fed-RL) approach to tackle (a) model complexities, unknown dynamical behaviors of IBR devices, (b) privacy issues regarding data sharing in multi-party-owned networked grids, and (2) transfers learned controls from simulation to hardware-in-the-loop test-bed, thereby bridging the gap between simulation and real world. With these multi-prong objectives, first, we formulate a reinforcement learning (RL) training setup generating episodic trajectories with adversaries (attack signal) injected at the primary controllers of the grid forming (GFM) inverters where RL agents (or controllers) are being trained to mitigate the injected attacks. For networked microgrids, the horizontal Fed-RL method involving distinct independent environments is not appropriate, leading us to develop vertical variant Federated Soft Actor-Critic (FedSAC) algorithm to grasp the interconnected dynamics of networked microgrid. Next, utilizing OpenAI Gym interface, we built a custom simulation set-up in GridLAB-D/HELICS co-simulation platform, named Resilient RL Co-simulation (ResRLCoSIM), to train the RL agents with IEEE 123-bus benchmark test systems comprising 3 interconnected microgrids. Finally, the learned policies in simulation world are transferred to the real-time hardware-in-the-loop test-bed set-up developed using high-fidelity Hypersim platform. Experiments show that the simulator-trained RL controllers produce convincing results with the real-time test-bed set-up, validating the minimization of sim-to-real gap.

Figures (7)

• Figure 1: Resilient RL Co-simulation (ResRLCoSIM) Platform for Microgrids
• Figure 2: Schematic representation of the Python-Hypersim test-bed co-simulation platform.
• Figure 3: (a) Networked Microgird with individual control agents, (b) Fed-RL framework
• Figure 4: (a) Agent-wise reward plot for Federated SAC training for different seeds, (b) Comparison of Federated SAC and Multi-agent Decentralized SAC, (c) Agent-wise accumulated rewards.
• Figure 5: Testing performance on the grid-forming inverter terminals