Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning for Sustainable Energy: A Survey

Koen Ponse, Felix Kleuker, Márton Fejér, Álvaro Serra-Gómez, Aske Plaat, Thomas Moerland

TL;DR

This survey addresses reinforcement learning for sustainable energy across the entire energy chain—generation, storage, consumption, and grids—arguing that RL's sequential decision-making capabilities can address intermittency, storage optimization, demand management, and grid stability. It provides an integrated taxonomy, surveys of RL basics, and a detailed review of applications by pillar, while highlighting overarching RL themes (multi-agent, offline, and safe RL) and the need for standardized benchmarks (e.g., SustainGym, CityLearn, L2RPN) to accelerate cross-disciplinary progress. The authors identify key challenges, including data scarcity, model uncertainty, safety constraints, and the gap between energy research and ML methodology, and propose future directions such as model-based and offline RL, graph-based approaches, and multi-objective/multi-scale methods. Overall, the paper argues that standardized benchmarks and closer collaboration between energy researchers and ML researchers are essential to realize RL's potential in enabling a scalable and reliable energy transition.

Abstract

The transition to sustainable energy is a key challenge of our time, requiring modifications in the entire pipeline of energy production, storage, transmission, and consumption. At every stage, new sequential decision-making challenges emerge, ranging from the operation of wind farms to the management of electrical grids or the scheduling of electric vehicle charging stations. All such problems are well suited for reinforcement learning, the branch of machine learning that learns behavior from data. Therefore, numerous studies have explored the use of reinforcement learning for sustainable energy. This paper surveys this literature with the intention of bridging both the underlying research communities: energy and machine learning. After a brief introduction of both fields, we systematically list relevant sustainability challenges, how they can be modeled as a reinforcement learning problem, and what solution approaches currently exist in the literature. Afterwards, we zoom out and identify overarching reinforcement learning themes that appear throughout sustainability, such as multi-agent, offline, and safe reinforcement learning. Lastly, we also cover standardization of environments, which will be crucial for connecting both research fields, and highlight potential directions for future work. In summary, this survey provides an extensive overview of reinforcement learning methods for sustainable energy, which may play a vital role in the energy transition.

Reinforcement Learning for Sustainable Energy: A Survey

TL;DR

This survey addresses reinforcement learning for sustainable energy across the entire energy chain—generation, storage, consumption, and grids—arguing that RL's sequential decision-making capabilities can address intermittency, storage optimization, demand management, and grid stability. It provides an integrated taxonomy, surveys of RL basics, and a detailed review of applications by pillar, while highlighting overarching RL themes (multi-agent, offline, and safe RL) and the need for standardized benchmarks (e.g., SustainGym, CityLearn, L2RPN) to accelerate cross-disciplinary progress. The authors identify key challenges, including data scarcity, model uncertainty, safety constraints, and the gap between energy research and ML methodology, and propose future directions such as model-based and offline RL, graph-based approaches, and multi-objective/multi-scale methods. Overall, the paper argues that standardized benchmarks and closer collaboration between energy researchers and ML researchers are essential to realize RL's potential in enabling a scalable and reliable energy transition.

Abstract

The transition to sustainable energy is a key challenge of our time, requiring modifications in the entire pipeline of energy production, storage, transmission, and consumption. At every stage, new sequential decision-making challenges emerge, ranging from the operation of wind farms to the management of electrical grids or the scheduling of electric vehicle charging stations. All such problems are well suited for reinforcement learning, the branch of machine learning that learns behavior from data. Therefore, numerous studies have explored the use of reinforcement learning for sustainable energy. This paper surveys this literature with the intention of bridging both the underlying research communities: energy and machine learning. After a brief introduction of both fields, we systematically list relevant sustainability challenges, how they can be modeled as a reinforcement learning problem, and what solution approaches currently exist in the literature. Afterwards, we zoom out and identify overarching reinforcement learning themes that appear throughout sustainability, such as multi-agent, offline, and safe reinforcement learning. Lastly, we also cover standardization of environments, which will be crucial for connecting both research fields, and highlight potential directions for future work. In summary, this survey provides an extensive overview of reinforcement learning methods for sustainable energy, which may play a vital role in the energy transition.
Paper Structure (13 sections, 4 equations, 7 figures, 3 tables)

This paper contains 13 sections, 4 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Areas of Sustainable Energy
  4. Reinforcement Learning Basics
  5. Applications of Reinforcement Learning in Sustainable Energy
  6. Generation
  7. Storage
  8. Consumption
  9. Electrical Grids
  10. Reinforcement Learning Challenges
  11. Benchmarks, baselines and performance metrics
  12. Discussion and Future Work
  13. Conclusion

Figures (7)

  • Figure 1: Potential development of the energy mix from today towards full sustainability (trajectories are not fully based on real projective data and are just for illustrative purposes)
  • Figure 2: Overview of energy systems and our taxonomy used throughout this survey. We separate sustainable energy systems in one of four pillars: generation, storage, consumption and transmission. Notably, transmission links all the various energy areas. This taxonomy will be used primarily in Sections \ref{['sec:3']} and \ref{['sec:4']}, where we briefly introduce each pillars, and dive deeper into the active research in each of the pillars, respectively.
  • Figure 3: Overview of sustainable power generation, the first pillar in our taxonomy.
  • Figure 4: Overview of sustainability challenges for reinforcement learning in consumption domain. Challenges lie in sub-fields related to buildings, EV(-charging), and highly specific industrial requirements.
  • Figure 5: Overview of electrical grid connectivity and control possibilities. Power stations (left), e.g. traditional plants, distributed energy resources, battery farms, hydrodams, etc., feed electricity into the network. High-voltage transmission lines (top) then route energy over long distances, facilitating transmission between generation sources and distribution substations. Along the way, the grid contains substations to ensure the safe routing and operation of electricity, serving as junction points where voltage levels are adjusted and power flow is managed. Distribution substations further reduce the voltage for distribution to end-users (right), such as households, commercial buildings, and industrial facilities. At these substations, electricity is directed to distribution transformers, which further adjust voltage levels for local distribution. Microgrids (bottom right) and controllers provide localized energy management solutions, effectively separating local control from the main grid control problem.
  • ...and 2 more figures