Heterogeneous Multi-Agent Proximal Policy Optimization for Power Distribution System Restoration

TL;DR

This work tackles the challenge of restoring large power distribution systems after outages by coordinating heterogeneous microgrids through a Heterogeneous-Agent Proximal Policy Optimization (HAPPO) framework. A centralized critic with decentralized actors enables stable, on-policy learning in highly coupled, structurally diverse environments, while a physics-informed OpenDSS simulator enforces constraints via differentiable penalties rather than action masking. The approach demonstrates superior restoration performance and reproducibility on IEEE 123-bus and 8500-node feeders, achieving around 92-96% of the system-wide generation cap under a 2400 kW limit with scalable, multi-seed learning. These results suggest that HARL-based, constraint-aware policy optimization is a promising path for real-time, large-scale distribution system restoration with practical heterogeneity and fidelity to physical laws.

Abstract

Restoring power distribution systems (PDS) after large-scale outages requires sequential switching operations that reconfigure feeder topology and coordinate distributed energy resources (DERs) under nonlinear constraints such as power balance, voltage limits, and thermal ratings. These challenges make conventional optimization and value-based RL approaches computationally inefficient and difficult to scale. This paper applies a Heterogeneous-Agent Reinforcement Learning (HARL) framework, instantiated through Heterogeneous-Agent Proximal Policy Optimization (HAPPO), to enable coordinated restoration across interconnected microgrids. Each agent controls a distinct microgrid with different loads, DER capacities, and switch counts, introducing practical structural heterogeneity. Decentralized actor policies are trained with a centralized critic to compute advantage values for stable on-policy updates. A physics-informed OpenDSS environment provides full power flow feedback and enforces operational limits via differentiable penalty signals rather than invalid action masking. The total DER generation is capped at 2400 kW, and each microgrid must satisfy local supply-demand feasibility. Experiments on the IEEE 123-bus and IEEE 8500-node systems show that HAPPO achieves faster convergence, higher restored power, and smoother multi-seed training than DQN, PPO, MAES, MAGDPG, MADQN, Mean-Field RL, and QMIX. Results demonstrate that incorporating microgrid-level heterogeneity within the HARL framework yields a scalable, stable, and constraint-aware solution for complex PDS restoration.

Figures (4)

• Figure 1: Overview of the proposed HAPPO-based HARL framework for multi-agent power distribution system restoration. Individual microgrid actors issue switching actions based on local observations, while a shared centralized critic computes value and advantage estimates from global states. OpenDSS opendss simulation feedback enables constraint-aware sequential policy updates.
• Figure 2: IEEE 123-bus distribution feeder used in the experiments. The system includes 26 controllable switches and five DERs (total generation 2400 kW), partitioned into five microgrid regions.
• Figure 3: Training performance of HAPPO on the IEEE 123-bus feeder across four random seeds. Left: cumulative reward. Right: restored active power.
• Figure 4: Training performance of HAPPO on the IEEE 8500-node feeder across five random seeds. Left: cumulative reward. Right: restored active power.