Reinforcement Learning-Based Co-Design and Operation of Chiller and Thermal Energy Storage for Cost-Optimal HVAC Systems

TL;DR

This work tackles the co-design of a fixed-capacity chiller and a TES system for cost-optimal HVAC under stochastic load and time-varying electricity prices. It casts hourly operation as a finite-horizon MDP with constrained action spaces and solves it via a maskable Deep Q-Network to learn cost-efficient, feasible control policies. The authors couple this operational policy with a 30-year life-cycle cost model to identify the cost-optimal sizing, demonstrating that a balanced design (700 kWh extsubscript{th} chiller and 1500 kWh extsubscript{th} TES) minimizes LCC while avoiding load losses. They show RL-based control reduces operating cost relative to baseline policies and expands feasible sizing configurations, highlighting the practical impact for energy-system design and operation. The approach underscores the value of reinforcement learning as a computational tool for estimating optimal operational cost in co-design problems with complex, constrained dynamics.

Abstract

We study the joint operation and sizing of cooling infrastructure for commercial HVAC systems using reinforcement learning, with the objective of minimizing life-cycle cost over a 30-year horizon. The cooling system consists of a fixed-capacity electric chiller and a thermal energy storage (TES) unit, jointly operated to meet stochastic hourly cooling demands under time-varying electricity prices. The life-cycle cost accounts for both capital expenditure and discounted operating cost, including electricity consumption and maintenance. A key challenge arises from the strong asymmetry in capital costs: increasing chiller capacity by one unit is far more expensive than an equivalent increase in TES capacity. As a result, identifying the right combination of chiller and TES sizes, while ensuring zero loss-of-cooling-load under optimal operation, is a non-trivial co-design problem. To address this, we formulate the chiller operation problem for a fixed infrastructure configuration as a finite-horizon Markov Decision Process (MDP), in which the control action is the chiller part-load ratio (PLR). The MDP is solved using a Deep Q Network (DQN) with a constrained action space. The learned DQN RL policy minimizes electricity cost over historical traces of cooling demand and electricity prices. For each candidate chiller-TES sizing configuration, the trained policy is evaluated. We then restrict attention to configurations that fully satisfy the cooling demand and perform a life-cycle cost minimization over this feasible set to identify the cost-optimal infrastructure design. Using this approach, we determine the optimal chiller and thermal energy storage capacities to be 700 and 1500, respectively.

Figures (3)

• Figure 1: Illustration of Distributed HVAC System
• Figure 2: Control logic for TES-First Feasibility Policy (TFP)
• Figure 3: Architecture diagram of DQN