Degradation-Aware Frequency Regulation of a Heterogeneous Battery Fleet via Reinforcement Learning
Tanay Raghunandan Srinivasa, Vivek Deulkar, Jia Bhargava, Mohammad Hajiesmaili, Prashant Shenoy
TL;DR
The paper tackles degrading effects in a heterogeneous battery fleet providing frequency regulation by formulating the control problem as a degradation-aware Markov decision process. It introduces a dense proxy reward based on online switching-point dynamics to guide learning toward shallower cycles, paired with an Extreme Learning Machine-based feature map to scale to large state-action spaces. The method is evaluated on synthetic Markov signals and real PJM RegD traces, showing consistent reductions in rainflow-detected degradation compared with baseline policies. The work advances practical deployment of battery fleets for grid services by balancing regulation performance with long-term battery lifetime, and suggests future work to integrate tracking error and other aging factors into the learning framework.
Abstract
Battery energy storage systems are increasingly deployed as fast-responding resources for grid balancing services such as frequency regulation and for mitigating renewable generation uncertainty. However, repeated charging and discharging induces cycling degradation and reduces battery lifetime. This paper studies the real-time scheduling of a heterogeneous battery fleet that collectively tracks a stochastic balancing signal subject to per-battery ramp-rate and capacity constraints, while minimizing long-term cycling degradation. Cycling degradation is fundamentally path-dependent: it is determined by charge-discharge cycles formed by the state-of-charge (SoC) trajectory and is commonly quantified via rainflow cycle counting. This non-Markovian structure makes it difficult to express degradation as an additive per-time-step cost, complicating classical dynamic programming approaches. We address this challenge by formulating the fleet scheduling problem as a Markov decision process (MDP) with constrained action space and designing a dense proxy reward that provides informative feedback at each time step while remaining aligned with long-term cycle-depth reduction. To scale learning to large state-action spaces induced by fine-grained SoC discretization and asymmetric per-battery constraints, we develop a function-approximation reinforcement learning method using an Extreme Learning Machine (ELM) as a random nonlinear feature map combined with linear temporal-difference learning. We evaluate the proposed approach on a toy Markovian signal model and on a Markovian model trained from real-world regulation signal traces obtained from the University of Delaware, and demonstrate consistent reductions in cycle-depth occurrence and degradation metrics compared to baseline scheduling policies.