Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimizing Power Grid Topologies with Reinforcement Learning: A Survey of Methods and Challenges

Erica van der Sar, Alessandro Zocca, Sandjai Bhulai

TL;DR

The paper surveys reinforcement learning approaches for power grid topology optimization, with a focus on the L2RPN benchmark suite and Grid2Op environments. It systematically categorizes design choices (algorithms, action/observation spaces, rewards, curricula, and rule-based heuristics) and provides a comparative numerical study on a standard 14-bus sandbox, highlighting when action-space reduction, robust rules, and high activation thresholds improve training and performance. Key findings indicate that while PPO-based agents can approach or exceed baselines under carefully designed reductions and curricula, theGreedy baseline often remains a strong competitor, underscoring the need for robust, scalable, and generalizable RL strategies. The authors advocate future work in imitation learning, off-policy methods, graph neural networks, and model-based planning to bridge the gap to real-world grid optimization and larger grids, offering practical guidance for researchers and operators. The work thus lays a foundation for advancing RL-driven power grid optimization by detailing open challenges, benchmarking practices, and concrete recommendations for future experiments.

Abstract

Power grid operation is becoming increasingly complex due to the rising integration of renewable energy sources and the need for more adaptive control strategies. Reinforcement Learning (RL) has emerged as a promising approach to power network control (PNC), offering the potential to enhance decision-making in dynamic and uncertain environments. The Learning To Run a Power Network (L2RPN) competitions have played a key role in accelerating research by providing standardized benchmarks and problem formulations, leading to rapid advancements in RL-based methods. This survey provides a comprehensive and structured overview of RL applications for power grid topology optimization, categorizing existing techniques, highlighting key design choices, and identifying gaps in current research. Additionally, we present a comparative numerical study evaluating the impact of commonly applied RL-based methods, offering insights into their practical effectiveness. By consolidating existing research and outlining open challenges, this survey aims to provide a foundation for future advancements in RL-driven power grid optimization.

Optimizing Power Grid Topologies with Reinforcement Learning: A Survey of Methods and Challenges

TL;DR

The paper surveys reinforcement learning approaches for power grid topology optimization, with a focus on the L2RPN benchmark suite and Grid2Op environments. It systematically categorizes design choices (algorithms, action/observation spaces, rewards, curricula, and rule-based heuristics) and provides a comparative numerical study on a standard 14-bus sandbox, highlighting when action-space reduction, robust rules, and high activation thresholds improve training and performance. Key findings indicate that while PPO-based agents can approach or exceed baselines under carefully designed reductions and curricula, theGreedy baseline often remains a strong competitor, underscoring the need for robust, scalable, and generalizable RL strategies. The authors advocate future work in imitation learning, off-policy methods, graph neural networks, and model-based planning to bridge the gap to real-world grid optimization and larger grids, offering practical guidance for researchers and operators. The work thus lays a foundation for advancing RL-driven power grid optimization by detailing open challenges, benchmarking practices, and concrete recommendations for future experiments.

Abstract

Power grid operation is becoming increasingly complex due to the rising integration of renewable energy sources and the need for more adaptive control strategies. Reinforcement Learning (RL) has emerged as a promising approach to power network control (PNC), offering the potential to enhance decision-making in dynamic and uncertain environments. The Learning To Run a Power Network (L2RPN) competitions have played a key role in accelerating research by providing standardized benchmarks and problem formulations, leading to rapid advancements in RL-based methods. This survey provides a comprehensive and structured overview of RL applications for power grid topology optimization, categorizing existing techniques, highlighting key design choices, and identifying gaps in current research. Additionally, we present a comparative numerical study evaluating the impact of commonly applied RL-based methods, offering insights into their practical effectiveness. By consolidating existing research and outlining open challenges, this survey aims to provide a foundation for future advancements in RL-driven power grid optimization.

Paper Structure

This paper contains 84 sections, 19 equations, 18 figures, 33 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Introduction to Power Grids
  4. From generation to consumption
  5. Topology Optimization
  6. Introduction to Reinforcement Learning
  7. Key concepts reinforcement learning
  8. Deep RL
  9. RL methods and terminology
  10. Model-based vs. model-free
  11. Value-based vs. policy-based methods
  12. On-policy vs. off-policy
  13. Problem Description
  14. Reinforcement Learning Framework for Power Grid Management
  15. Rules of the game
  16. ...and 69 more sections

Figures (18)

  • Figure 1: Schematic illustration of power distribution from generation sources to consumers.
  • Figure 2: Example of a transmission power grid represented as a directed graph. This example is part of the environment l2rpn_icaps_2021, available in Grid2Op (see \ref{['Sec:ProblemDescr']}).
  • Figure 3: An overview of the algorithms applied to L2RPN categorized by the features discussed in this section.
  • Figure 4: Example of a bus-splitting action by an agent and the effect on the environment. Some elements of substation 0 are assigned to bus 2, which results in a split of substation 0, and a consequent power flow redistribution.
  • Figure 5: Boxplot of the time steps survived for the baseline agents applied to the validation episodes.
  • ...and 13 more figures