Optimizing Energy Management of Smart Grid using Reinforcement Learning aided by Surrogate models built using Physics-informed Neural Networks

TL;DR

This work tackles the sample-efficiency challenge of reinforcement learning for Optimal Power Flow in smart grids by introducing Physics-Informed Neural Network (PINN) surrogates to replace costly simulators. It integrates PINN surrogates with PPO in the Gym-ANM framework and demonstrates a 50% reduction in policy-training time and a 10x speedup in inference, while maintaining or improving policy reliability. The study shows that PINN-based surrogates outperform purely data-driven models, particularly in long-horizon or changing-demand scenarios, due to their physics-consistent extrapolation. Overall, the approach enables faster, safer, and more scalable RL-based energy-management strategies for renewable-rich smart grids and sets the stage for applying this method to larger, more complex networks.

Abstract

Optimizing the energy management within a smart grids scenario presents significant challenges, primarily due to the complexity of real-world systems and the intricate interactions among various components. Reinforcement Learning (RL) is gaining prominence as a solution for addressing the challenges of Optimal Power Flow in smart grids. However, RL needs to iterate compulsively throughout a given environment to obtain the optimal policy. This means obtaining samples from a, most likely, costly simulator, which can lead to a sample efficiency problem. In this work, we address this problem by substituting costly smart grid simulators with surrogate models built using Phisics-informed Neural Networks (PINNs), optimizing the RL policy training process by arriving to convergent results in a fraction of the time employed by the original environment.

Figures (10)

• Figure 1: Architecture of our approach. There are two different methods for training the surrogate environment, used for either the PINN method or the data-driven methods. The bold dashed lines show the PINN training process, while the thin dashed line represents the data-driven process instead. Subsequently, the surrogate environment is used to train the RL policy, and after each policy update, the policy is tested against the original environment to get an accurate representation of the improvement of the policy.
• Figure 2: ANM6-Easy environment: an electrical network composed of a slack generator, $2$ renewable energy generators (a wind farm and a PV plant), $3$ passive loads (an industrial complex, a residential area, and an EV charging garage), and a storage unit.
• Figure 3: Generators Loss: Aggregate of KKT residuals femine2024kktinformed associated with generator-related KKT-NN. DES Loss: Aggregate of KKT residuals femine2024kktinformed associated with battery-related KKT-NN. Power Balance Loss: Residuals from power balance equations \ref{['eq:loss-power-balance']} associated with voltage-related PINN
• Figure 4: Comparison of inference times over 1000 transitions. The median time for PINN stands at $0.0021~\text{s}$, while the median for the original environment is $0.0194~\text{s}$.
• Figure 5: R$^2$ and MAE score comparison between the studied surrogate models over the two different datasets: Generative and Agent-based transitions. The R$^2$ is in linear scale while the MAE is in logarithmic scale. Clearly, the XGB models show the highest accuracy and the smallest error, followed closely by the PINN model. However, despite their good overall accuracy, most surrogates do not perform well when used as an RL training environment, excluding the PINN model.