Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement learning based demand charge minimization using energy storage

Lucas Weber, Ana Bušić, Jiamin Zhu

Abstract

Utilities have introduced demand charges to encourage customers to reduce their demand peaks, since a high peak may cause very high costs for both the utility and the consumer. We herein study the bill minimization problem for customers equipped with an energy storage device and a self-owned renewable energy production. A model-free reinforcement learning algorithm is carefully designed to reduce both the energy charge and the demand charge of the consumer. The proposed algorithm does not need forecasting models for the energy demand and the renewable energy production. The resulting controller can be used online, and progressively improved with newly gathered data. The algorithm is validated on real data from an office building of IFPEN Solaize site. Numerical results show that our algorithm can reduce electricity bills with both daily and monthly demand charges.

Reinforcement learning based demand charge minimization using energy storage

Abstract

Utilities have introduced demand charges to encourage customers to reduce their demand peaks, since a high peak may cause very high costs for both the utility and the consumer. We herein study the bill minimization problem for customers equipped with an energy storage device and a self-owned renewable energy production. A model-free reinforcement learning algorithm is carefully designed to reduce both the energy charge and the demand charge of the consumer. The proposed algorithm does not need forecasting models for the energy demand and the renewable energy production. The resulting controller can be used online, and progressively improved with newly gathered data. The algorithm is validated on real data from an office building of IFPEN Solaize site. Numerical results show that our algorithm can reduce electricity bills with both daily and monthly demand charges.
Paper Structure (25 sections, 20 equations, 7 figures, 2 tables)

This paper contains 25 sections, 20 equations, 7 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. RL background
  3. Reinforcement learning
  4. Evaluation step
  5. Policy improvement step
  6. Application to bill minimization problems
  7. Bill minimization problems
  8. Markov decision process
  9. System model
  10. State and action spaces
  11. Cost function design
  12. Search for an optimal policy
  13. Problem decomposition
  14. Decomposition-based PI algorithm
  15. Choice of initial policy
  16. ...and 10 more sections

Figures (7)

  • Figure 1: Sketch of the system
  • Figure 2: Swarm plot representing the daily bill reductions on the test set when using our agents versus no battery. Each color represents one month.
  • Figure 3: Swarm plot representing the bill reductions on the test set for the heuristic, agents $A_5$ and agent $A_6$.
  • Figure 4: Days with small and big bill reduction. Top figures show the power generation, the electricity demand and the battery charge. Bottom figures represent $P_{meter}$ with and without battery. Gray areas are on-peak hours.
  • Figure 5: Swarm plot representing the daily bill reductions on the test set for the heuristic, agent $A_7$ and agent $A_8$.
  • ...and 2 more figures

Theorems & Definitions (3)

  • Remark 1
  • Remark 2
  • Remark 3