Reinforcement Learning (RL) Meets Urban Climate Modeling: Investigating the Efficacy and Impacts of RL-Based HVAC Control

TL;DR

This study presents a framework that couples reinforcement learning with an urban climate model (CLMU) and a building energy model (BEM) to evaluate RL-based HVAC control across cities with diverse background climates. By using a Python surrogate for offline RL training and online coupling with CLMU, the work assesses impacts on indoor climate, local urban climate, and the transferability of learned policies between cities. Key findings show climate-dependent variability in reward and climate impacts, with SAC delivering the best performance among tested algorithms and transferability influenced by temperature variability and latitude. The results highlight the need for climate-aware, city-specific RL strategies and suggest city-to-city learning could enhance deployment, while acknowledging limitations of the surrogate approach and the simplified urban canyon modeling.

Abstract

Reinforcement learning (RL)-based heating, ventilation, and air conditioning (HVAC) control has emerged as a promising technology for reducing building energy consumption while maintaining indoor thermal comfort. However, the efficacy of such strategies is influenced by the background climate and their implementation may potentially alter both the indoor climate and local urban climate. This study proposes an integrated framework combining RL with an urban climate model that incorporates a building energy model, aiming to evaluate the efficacy of RL-based HVAC control across different background climates, impacts of RL strategies on indoor climate and local urban climate, and the transferability of RL strategies across cities. Our findings reveal that the reward (defined as a weighted combination of energy consumption and thermal comfort) and the impacts of RL strategies on indoor climate and local urban climate exhibit marked variability across cities with different background climates. The sensitivity of reward weights and the transferability of RL strategies are also strongly influenced by the background climate. Cities in hot climates tend to achieve higher rewards across most reward weight configurations that balance energy consumption and thermal comfort, and those cities with more varying atmospheric temperatures demonstrate greater RL strategy transferability. These findings underscore the importance of thoroughly evaluating RL-based HVAC control strategies in diverse climatic contexts. This study also provides a new insight that city-to-city learning will potentially aid the deployment of RL-based HVAC control.

Figures (19)

• Figure 1: Cities selected for this study.
• Figure 2: Schematic diagrams of RL training with surrogate and coupling with CLMU. $T_\text{a}$ is the urban canopy air temperature; $T_\text{b}$ is the building indoor air temperature; $H_\text{wh}$ is the heat flux due to inefficiencies in heating and air conditioning systems, as well as energy losses during the conversion of primary energy sources into usable end-use energy; $H_\text{ac}$ is heat flux extracted from the building's interior by the air conditioning system. Both $H_\text{wh}$ and $H_\text{ac}$ can be put into the canyon floor as sensible heat in CLMU simulations.
• Figure 3: Mean reward difference in RL case and baselines (CLMU default setting) across cities and monthly profiles. The size of the marker indicates the standard error.
• Figure 4: Monthly profile of difference of indoor air temperatures of buildings (TBUILD$_\text{diff}$) in RL case and CLMU default case across cities.
• Figure 5: Monthly profile of outdoor air temperature difference (TA$_\text{diff}$) in RL case and CLMU default case across cities.