Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Reinforcement Learning Approach for the Continuous Electricity Market of Germany: Trading from the Perspective of a Wind Park Operator

Malte Lehna, Björn Hoppmann, René Heinrich, Christoph Scholz

TL;DR

A novel autonomous trading approach based on Deep Reinforcement Learning (DRL) algorithms as a possible solution to the intraday electricity markets and is able to outperform multiple baselines with at least 45.24% improvement, showing the advantage of the DRL algorithm.

Abstract

With the rising extension of renewable energies, the intraday electricity markets have recorded a growing popularity amongst traders as well as electric utilities to cope with the induced volatility of the energy supply. Through their short trading horizon and continuous nature, the intraday markets offer the ability to adjust trading decisions from the day-ahead market or reduce trading risk in a short-term notice. Producers of renewable energies utilize the intraday market to lower their forecast risk, by modifying their provided capacities based on current forecasts. However, the market dynamics are complex due to the fact that the power grids have to remain stable and electricity is only partly storable. Consequently, robust and intelligent trading strategies are required that are capable to operate in the intraday market. In this work, we propose a novel autonomous trading approach based on Deep Reinforcement Learning (DRL) algorithms as a possible solution. For this purpose, we model the intraday trade as a Markov Decision Problem (MDP) and employ the Proximal Policy Optimization (PPO) algorithm as our DRL approach. A simulation framework is introduced that enables the trading of the continuous intraday price in a resolution of one minute steps. We test our framework in a case study from the perspective of a wind park operator. We include next to general trade information both price and wind forecasts. On a test scenario of German intraday trading results from 2018, we are able to outperform multiple baselines with at least 45.24% improvement, showing the advantage of the DRL algorithm. However, we also discuss limitations and enhancements of the DRL agent, in order to increase the performance in future works.

A Reinforcement Learning Approach for the Continuous Electricity Market of Germany: Trading from the Perspective of a Wind Park Operator

TL;DR

A novel autonomous trading approach based on Deep Reinforcement Learning (DRL) algorithms as a possible solution to the intraday electricity markets and is able to outperform multiple baselines with at least 45.24% improvement, showing the advantage of the DRL algorithm.

Abstract

With the rising extension of renewable energies, the intraday electricity markets have recorded a growing popularity amongst traders as well as electric utilities to cope with the induced volatility of the energy supply. Through their short trading horizon and continuous nature, the intraday markets offer the ability to adjust trading decisions from the day-ahead market or reduce trading risk in a short-term notice. Producers of renewable energies utilize the intraday market to lower their forecast risk, by modifying their provided capacities based on current forecasts. However, the market dynamics are complex due to the fact that the power grids have to remain stable and electricity is only partly storable. Consequently, robust and intelligent trading strategies are required that are capable to operate in the intraday market. In this work, we propose a novel autonomous trading approach based on Deep Reinforcement Learning (DRL) algorithms as a possible solution. For this purpose, we model the intraday trade as a Markov Decision Problem (MDP) and employ the Proximal Policy Optimization (PPO) algorithm as our DRL approach. A simulation framework is introduced that enables the trading of the continuous intraday price in a resolution of one minute steps. We test our framework in a case study from the perspective of a wind park operator. We include next to general trade information both price and wind forecasts. On a test scenario of German intraday trading results from 2018, we are able to outperform multiple baselines with at least 45.24% improvement, showing the advantage of the DRL algorithm. However, we also discuss limitations and enhancements of the DRL agent, in order to increase the performance in future works.
Paper Structure (36 sections, 6 equations, 8 figures, 5 tables)

This paper contains 36 sections, 6 equations, 8 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Overview
  3. Related Work
  4. Research Contribution
  5. Methodology
  6. Reinforcement Learning as Markov Decision Process
  7. Optimization Algorithm
  8. Hyperparameter Optimization
  9. Research Design
  10. General Research Setting
  11. The German Continuous Intraday Electricity Market
  12. Restriction for the Trading Environment
  13. Detailed Description of the Environment
  14. Episodes
  15. Action Space
  16. ...and 21 more sections

Figures (8)

  • Figure 1: Visualization of the -Framework with environment and agent. Based on the state $s_t$ of the environment, the agent computes the action $a_t$. The action is returned, the reward $r_{t}$ calculated and saved for the optimization. Thereafter, the environment transitions to the new state $s_{t+1}$.
  • Figure 2: Exemplary neural network of the approach. The hidden layers (green) pass the state $s_t$ and reward $r_t$ information to the policy layer (light blue). Thereafter, an action $a_t$ is chosen based on the policy.
  • Figure 3: Overview over the trading stages of an product with delivery at day $d$ and time $t$. The trading starts at 3.00pm on the previous day $d-1$ and consist of national and cross-border trading. One hour the cross-border trading stops, leaving national trading, and half an hour only trading in the control area is allowed.
  • Figure 4: Visualization of the average number of trades per product, aggregated to minutely values, c.f. scholz2020towards. The left axis depicts the aggregated volume in MWh, while the right axis displays the number of trades in the minute. Regarding the case study, the left vertical line (black) shows the beginning of our trading interval, while the right vertical line (red) shows the end of the trading interval.
  • Figure 5: Plot of the 2018 German electricity price. Each product is visualized with its average, maximum and minimum value of the price. The training set is from May 2018 till mid of September, while the testing set is the remaining September (red).
  • ...and 3 more figures