Offline Reinforcement Learning for Microgrid Voltage Regulation

TL;DR

The paper addresses voltage regulation in PV-penetrated microgrids where online experimentation is unsafe or costly. It employs offline reinforcement learning, evaluating BCQ and CQL on a distflow-based IEEE 33-bus microgrid and training solely on pre-collected transitions. The results show CQL generally outperforms BCQ across datasets of varying quality due to its conservative Q-value estimation, improving robustness and stability. This demonstrates that Offline RL can yield safe, effective voltage-control policies without live interaction and offers a path toward scaling to larger, more complex grids.

Abstract

This paper presents a study on using different offline reinforcement learning algorithms for microgrid voltage regulation with solar power penetration. When environment interaction is unviable due to technical or safety reasons, the proposed approach can still obtain an applicable model through offline-style training on a previously collected dataset, lowering the negative impact of lacking online environment interactions. Experiment results on the IEEE 33-bus system demonstrate the feasibility and effectiveness of the proposed approach on different offline datasets, including the one with merely low-quality experience.

Figures (6)

• Figure 1: The basic idea of offline reinforcement learning.
• Figure 2: The episode returns across datasets.
• Figure 3: Performance comparison of BCQ and CQL across three datasets: Totally Controllable Ratio (top row) and Average Voltage Deviation (bottom row).
• Figure 4: The IEEE 33-bus system with solar PV penetration.
• Figure 5: Comparison of the VAE total loss across datasets.