Quantum-Enhanced Reinforcement Learning for Accelerating Newton-Raphson Convergence with Ising Machines: A Case Study for Power Flow Analysis

TL;DR

The paper tackles Newton-Raphson convergence issues in power flow analysis under poor initialization and stressed grid conditions. It proposes a reinforcement learning (RL) approach to optimize NR initialization, and couples it with a quantum-enhanced RL environment update that reframes voltage adjustments as a quadratic unconstrained binary optimization (QUBO) problem solved by Ising machines. The main contributions include training PPO-based agents to refine complex voltage initializations, demonstrating scalability from a 4-bus to a 14-bus system, and integrating quantum/digital annealing to efficiently explore large action spaces, reducing NR iterations and improving robustness. The results indicate substantial speedups and robustness gains, with QA and QIIO hardware delivering competitive performance and highlighting a practical pathway for combining classical solvers with quantum-inspired optimization in PF analysis.

Abstract

The Newton-Raphson (NR) method is widely used for solving power flow (PF) equations due to its quadratic convergence. However, its performance deteriorates under poor initialization or extreme operating scenarios, e.g., high levels of renewable energy penetration. Traditional NR initialization strategies often fail to address these challenges, resulting in slow convergence or even divergence. We propose the use of reinforcement learning (RL) to optimize the initialization of NR, and introduce a novel quantum-enhanced RL environment update mechanism to mitigate the significant computational cost of evaluating power system states over a combinatorially large action space at each RL timestep by formulating the voltage adjustment task as a quadratic unconstrained binary optimization problem. Specifically, quantum/digital annealers are integrated into the RL environment update to evaluate state transitions using a problem Hamiltonian designed for PF. Results demonstrate significant improvements in convergence speed, a reduction in NR iteration counts, and enhanced robustness under different operating conditions.

Figures (9)

• Figure 1: A visualized representation of a Reinforcement Learning procedure. At each timestep $t$, the agent observes state $s_t$ and performs action $a_t$ in the environment. The environment then transitions with transition dynamics $P$ into the next state $s_{t+1}$ and receives reward $r_{t+1}$.
• Figure 2: Schematic representation of the 4-bus test system. The system includes a slack bus and three load buses.
• Figure 3: Basin of attraction for initial values at load buses for the 4-bus test system. Each subplot shows the number of NR iterations (with the heatmap) needed to perform PF analysis for the given voltage magnitude ($p.u.$) and phase angle ($degrees$).
• Figure 4: Training trajectory of the RL agent for the 4-bus test system. The graph shows the evolution of episode reward and the number of NR iterations over $1.0e5$ timesteps.
• Figure 5: Training trajectory of the RL agent for the 14-bus test system. The graph shows the evolution of episode reward and (b) the number of NR iterations over $2.5e4$ timesteps.