SINERGYM -- A virtual testbed for building energy optimization with Reinforcement Learning

TL;DR

This paper introduces Sinergym, an open-source virtual testbed that enables reinforcement learning-based Building Energy Optimization (BEO) by integrating EnergyPlus simulations with a Gymnasium-compatible interface. It provides a flexible, scalable platform with customizable building models, weather variability, reward definitions, wrappers, controllers, and benchmarking to support large-scale experimentation and reproducible comparisons. Through examples including default, rule-based, and deep RL controllers, and a hyperparameter optimization study, the authors demonstrate that DRL—especially PPO—can achieve meaningful energy savings while maintaining comfort. The work advances digital twin-inspired building operations by offering a standardized, well-documented toolchain for training, evaluating, and comparing RL-based control strategies, with future plans for graphical interfaces, additional co-simulation engines, and broader benchmarking efforts.

Abstract

Simulation has become a crucial tool for Building Energy Optimization (BEO) as it enables the evaluation of different design and control strategies at a low cost. Machine Learning (ML) algorithms can leverage large-scale simulations to learn optimal control from vast amounts of data without supervision, particularly under the Reinforcement Learning (RL) paradigm. Unfortunately, the lack of open and standardized tools has hindered the widespread application of ML and RL to BEO. To address this issue, this paper presents Sinergym, an open-source Python-based virtual testbed for large-scale building simulation, data collection, continuous control, and experiment monitoring. Sinergym provides a consistent interface for training and running controllers, predefined benchmarks, experiment visualization and replication support, and comprehensive documentation in a ready-to-use software library. This paper 1) highlights the main features of Sinergym in comparison to other existing frameworks, 2) describes its basic usage, and 3) demonstrates its applicability for RL-based BEO through several representative examples. By integrating simulation, data, and control, Sinergym supports the development of intelligent, data-driven applications for more efficient and responsive building operations, aligning with the objectives of digital twin technology.

Figures (9)

• Figure 1: Sinergym's general architecture, depicting its three main layers: communication, middleware and simulator
• Figure 2: Overview of the Sinergym workflow
• Figure 3: Ornstein-Uhlenbeck noise applied to a mixed weather dataset (New York) with different values for $\sigma$, $\mu$ and $\tau$
• Figure 4: Example of intermediate values computed by the reward function
• Figure 5: Indoor air temperatures achieved by each controller for a 1-year evaluation period in 2ZoneDataCenterHVAC with mixed weather. The red and blue dotted straight lines indicate the comfort range. Indoor air temperatures are shown in colour, while outdoor temperatures are shown in grey.