Continual Reinforcement Learning for HVAC Systems Control: Integrating Hypernetworks and Transfer Learning

TL;DR

This work tackles the dual challenge of sample inefficiency and limited generalization in reinforcement learning for HVAC control. It proposes a model-based reinforcement learning framework that uses Hypernetworks to continuously learn environment dynamics across tasks with different action spaces, enabling efficient synthetic rollouts in a Dyna-style loop. The approach demonstrates backward transfer after training on a second task, and achieves rapid convergence on the first task with minimal retraining, effectively mitigating catastrophic forgetting and outperforming model-free baselines in sample efficiency. The results suggest significant practical impact for reducing energy use and operating costs in building management, supporting broader sustainability goals.

Abstract

Buildings with Heating, Ventilation, and Air Conditioning (HVAC) systems play a crucial role in ensuring indoor comfort and efficiency. While traditionally governed by physics-based models, the emergence of big data has enabled data-driven methods like Deep Reinforcement Learning (DRL). However, Reinforcement Learning (RL)-based techniques often suffer from sample inefficiency and limited generalization, especially across varying HVAC systems. We introduce a model-based reinforcement learning framework that uses a Hypernetwork to continuously learn environment dynamics across tasks with different action spaces. This enables efficient synthetic rollout generation and improved sample usage. Our approach demonstrates strong backward transfer in a continual learning setting after training on a second task, minimal fine-tuning on the first task allows rapid convergence within just 5 episodes and thus outperforming Model Free Reinforcement Learning (MFRL) and effectively mitigating catastrophic forgetting. These findings have significant implications for reducing energy consumption and operational costs in building management, thus supporting global sustainability goals. Keywords: Deep Reinforcement Learning, HVAC Systems Control, Hypernetworks, Transfer and Continual Learning, Catastrophic Forgetting

Figures (8)

• Figure 1: Hypernetwork architecture
• Figure 2: Training Loop
• Figure 3: Stage 1 January test results
• Figure 4: Stage 1 April test results
• Figure 5: Stage 2 January test results