A Reinforcement Learning Approach for Optimal Control in Microgrids

TL;DR

The paper tackles optimal microgrid energy management under renewable uncertainty and dynamic market prices by formulating the tertiary control problem as an MDP and solving it with a PPO-based reinforcement learning agent augmented by a digital twin for battery degradation. The agent observes a rich state that includes SoC, temperature, forecasts, and time features, and selects a continuous action that splits net power between the battery and the grid while respecting physical constraints. Experimental validation on real Italian data shows RL$^*$ achieves the best cumulative reward and robust performance across market and degradation scenarios, outperforming rule-based baselines and prior RL benchmarks. The work demonstrates practical viability for intelligent MG management and motivates future multi-agent extensions for cooperative trading across interconnected MGs.

Abstract

The increasing integration of renewable energy sources (RESs) is transforming traditional power grid networks, which require new approaches for managing decentralized energy production and consumption. Microgrids (MGs) provide a promising solution by enabling localized control over energy generation, storage, and distribution. This paper presents a novel reinforcement learning (RL)-based methodology for optimizing microgrid energy management. Specifically, we propose an RL agent that learns optimal energy trading and storage policies by leveraging historical data on energy production, consumption, and market prices. A digital twin (DT) is used to simulate the energy storage system dynamics, incorporating degradation factors to ensure a realistic emulation of the analysed setting. Our approach is validated through an experimental campaign using real-world data from a power grid located in the Italian territory. The results indicate that the proposed RL-based strategy outperforms rule-based methods and existing RL benchmarks, offering a robust solution for intelligent microgrid management.

Figures (8)

• Figure 1: Schema of the interactions between core MG elements. Single-edge arrows indicate a unidirectional positive exchange, while double-edge arrows represent bidirectional ones.
• Figure 2: Cumulative average empirical reward.
• Figure 3: Gaps for the trading (a) and degradation (b) components of the reward w.r.t. the 50-50 policy.
• Figure 4: Action vs. demand during a test profile run.
• Figure 5: Boxplot representing the empirical return across the $28$ validation profiles. Values above the plot represent the return of each method averaged over test profiles.