A Safe and Data-efficient Model-based Reinforcement Learning System for HVAC Control

TL;DR

CLUE optimizes HVAC operations by integrating a Gaussian process (GP) model to model building dynamics with uncertainty awareness, and incorporates these uncertainty estimates into a model predictive path integral (MPPI) algorithm, enabling the selection of safe, energy-efficient control actions.

Abstract

Model-Based Reinforcement Learning (MBRL) has been widely studied for Heating, Ventilation, and Air Conditioning (HVAC) control in buildings. One of the critical challenges is the large amount of data required to effectively train neural networks for modeling building dynamics. This paper presents CLUE, an MBRL system for HVAC control in buildings. CLUE optimizes HVAC operations by integrating a Gaussian Process (GP) model to model building dynamics with uncertainty awareness. CLUE utilizes GP to predict state transitions as Gaussian distributions, effectively capturing prediction uncertainty and enhancing decision-making under sparse data conditions. Our approach employs a meta-kernel learning technique to efficiently set GP kernel hyperparameters using domain knowledge from diverse buildings. This drastically reduces the data requirements typically associated with GP models in HVAC applications. Additionally, CLUE incorporates these uncertainty estimates into a Model Predictive Path Integral (MPPI) algorithm, enabling the selection of safe, energy-efficient control actions. This uncertainty-aware control strategy evaluates and selects action trajectories based on their predicted impact on energy consumption and human comfort, optimizing operations even under uncertain conditions. Extensive simulations in a five-zone office building demonstrate that CLUE reduces the required training data from hundreds of days to just seven while maintaining robust control performance. It reduces comfort violations by an average of 12.07% compared to existing MBRL methods, without compromising on energy efficiency.

Figures (12)

• Figure 1: Model errors are $>10\times$ higher in data-sparse regions vs. data-dense regions
• Figure 2: Model error distribution vs. training data set size
• Figure 3: CDF of the distances between model errors for each training set size
• Figure 4: KDE of the collected training data used for the dynamics model in MBRL-based HVAC control.
• Figure 5: Overview of the proposed CLUE system. The framework integrates two key components: a dynamic model of building systems and a control mechanism based on MPPI control.