Attentive Convolutional Deep Reinforcement Learning for Optimizing Solar-Storage Systems in Real-Time Electricity Markets

TL;DR

This work tackles the challenge of optimizing bids for a co-located solar PV farm and battery energy storage system in real-time electricity markets, aiming to both reduce solar curtailment and generate revenue from energy arbitrage. It introduces AC-DRL, a deep reinforcement learning framework that decouples bidding into two Markov decision processes and employs a stacked attention mechanism with multi-grained feature convolution to extract feature correlations for decision making. Experiments on real-world Australian NEM data show AC-DRL outperforms MPC-based and DRL baselines in total revenue and curtailment management, with notable gains for both Solar Farm and BESS components and substantial curtailment reductions. The results provide practical insights into BESS behavior under price dynamics, such as buy-low-sell-high arbitrage during daytime and increased curtailment absorption when curtailments are likely or prices spike, highlighting the method’s potential to improve the economic viability of solar-battery co-locations.

Abstract

This paper studies the synergy of solar-battery energy storage system (BESS) and develops a viable strategy for the BESS to unlock its economic potential by serving as a backup to reduce solar curtailments while also participating in the electricity market. We model the real-time bidding of the solar-battery system as two Markov decision processes for the solar farm and the BESS, respectively. We develop a novel deep reinforcement learning (DRL) algorithm to solve the problem by leveraging attention mechanism (AC) and multi-grained feature convolution to process DRL input for better bidding decisions. Simulation results demonstrate that our AC-DRL outperforms two optimization-based and one DRL-based benchmarks by generating 23%, 20%, and 11% higher revenue, as well as improving curtailment responses. The excess solar generation can effectively charge the BESS to bid in the market, significantly reducing solar curtailments by 76% and creating synergy for the solar-battery system to be more viable.

Figures (11)

• Figure 1: The system model paradigm.
• Figure 2: The detailed system configuration.
• Figure 3: The AC-DRL framework.
• Figure 4: The inner structure of one MHCA with two heads.
• Figure 5: Evaluation revenue comparisons of the DMPC benchmark, the SMPC benchmark, the MLP-DDPG benchmark, and our AC-DDPG.