Curriculum-based Reinforcement Learning for Distribution System Critical Load Restoration

TL;DR

This work tackles critical load restoration (CLR) in distribution systems under renewable forecast uncertainty by marrying reinforcement learning with curriculum learning (CL). A two-stage CL framework first trains on a simpler CLR problem with perfect forecasts, then transfers knowledge to a full problem incorporating imperfect renewable forecasts, yielding policies that converge to high performance and exhibit robustness to forecast errors. The RL policies are shown to outperform two MPC baselines, particularly as forecast error grows, and to scale to larger networks with manageable computational demands. By leveraging nonlinear OpenDSS power-flow modeling and a structured state-action-reward design, the approach achieves fast online control with improved resilience, suggesting significant potential for grid-operations under uncertainty.

Abstract

This paper focuses on the critical load restoration problem in distribution systems following major outages. To provide fast online response and optimal sequential decision-making support, a reinforcement learning (RL) based approach is proposed to optimize the restoration. Due to the complexities stemming from the large policy search space, renewable uncertainty, and nonlinearity in a complex grid control problem, directly applying RL algorithms to train a satisfactory policy requires extensive tuning to be successful. To address this challenge, this paper leverages the curriculum learning (CL) technique to design a training curriculum involving a simpler steppingstone problem that guides the RL agent to learn to solve the original hard problem in a progressive and more effective manner. We demonstrate that compared with direct learning, CL facilitates controller training to achieve better performance. To study realistic scenarios where renewable forecasts used for decision-making are in general imperfect, the experiments compare the trained RL controllers against two model predictive controllers (MPCs) using renewable forecasts with different error levels and observe how these controllers can hedge against the uncertainty. Results show that RL controllers are less susceptible to forecast errors than the baseline MPCs and can provide a more reliable restoration process.

Figures (11)

• Figure 1: Example of synthetic forecasts of wind generation with $\epsilon_T=0.1$. Top: Sampled 500 random drifts over time generated based on Proposition \ref{['proposition-1']}, i.e., $\sum_{i=0}^t \iota_i$. Middle: Corresponding synthetic forecasts (all normalized), i.e., $\Gamma_{[0,1]}(p_{k}^r /p^{r,\text{max}} + \sum_{i=1}^k \iota_i )$, given the 500 random drifts. Black curve is the actual generation $\mathbf{p}_{1:T}^r$, and the synthetic forecasts $\mathbf{\hat{p}}_{1:T}^r$ could be any one of the blue curves, the dashed line shows one specific over-forecast sample. Bottom: Red curve shows $\mathbf{\hat{p}}_{t:T}^r$ at a point and the trailing faded blue curves are forecasts at previous steps, i.e., $\mathbf{\hat{p}}_{t-i:T}^r, i \in \{1, ..., 5\}$. It is similar for PV synthetic forecasts generation.
• Figure 2: Illustration of scenario sampling and synthetic forecasts generation. Top: The renewable generation of one DER draxl2015wind, denoted as $\mathcal{P}^r \subset \mathcal{P}^\mathcal{R}$. Lower left: The actual renewable generation profile sampled for one episode, i.e., $\mathbf{p}^r_{1:T} \sim \mathcal{P}^r$; Lower right: This subplot highlights the relationship among $\mathbf{p}^r_{1:T} \in \mathbb{R}^{T}$, $\mathbf{\hat{p}}_{t:T}^r \in \mathbb{R}^{T-t+1}$ (updated forecasts at step $t$), $\mathbf{\hat{p}}_{t}^r \in \mathbb{R}^{k/\tau}$ (updated $k$-hour lookahead forecasts to be used in RL state \ref{['eq-renewable-forecasts']}) and $\mathbf{p}_{t}^r \in \mathbb{R}^{k/\tau}$ ($k$-hour lookahead perfect forecasts to be used in \ref{['eq-stage-I-state-definition']} in Section \ref{['subsec-cl-framework']}).
• Figure 3: Curriculum learning framework overview.
• Figure 4: A testing system modified from the IEEE 13-bus system.
• Figure 5: Learning environment for the CLR problem. Using the "reset" and "step" interfaces, the RL agent can sample different $T$-step control scenarios and explore and learn the optimal control strategy.

Theorems & Definitions (1)

• Proposition 1