Data Center Cooling System Optimization Using Offline Reinforcement Learning

TL;DR

Data centers incur substantial energy use, with cooling accounting for a large portion of power; the paper formulates the control problem as a Markov decision process with states $\mathcal{S}$, actions $\mathcal{A}$, transition $T$, reward $r$, and discount factor $\gamma$. It introduces a physics-informed offline RL framework built on a T-symmetry enforced thermal dynamics model (TTDM) and a graph neural encoder that learns latent representations; policy optimization occurs in latent space with a TD3+BC-style objective and a T-symmetry regularizer, enabling safe learning from limited data. In real-world deployment, the approach achieves 14–21% energy savings in production DC cooling without safety violations over 2000 hours and is complemented by a small-scale testbed with ablations. These results demonstrate offline RL's potential for data-limited, safety-critical industrial control and motivate broader adoption beyond toy simulators.

Abstract

The recent advances in information technology and artificial intelligence have fueled a rapid expansion of the data center (DC) industry worldwide, accompanied by an immense appetite for electricity to power the DCs. In a typical DC, around 30~40% of the energy is spent on the cooling system rather than on computer servers, posing a pressing need for developing new energy-saving optimization technologies for DC cooling systems. However, optimizing such real-world industrial systems faces numerous challenges, including but not limited to a lack of reliable simulation environments, limited historical data, and stringent safety and control robustness requirements. In this work, we present a novel physics-informed offline reinforcement learning (RL) framework for energy efficiency optimization of DC cooling systems. The proposed framework models the complex dynamical patterns and physical dependencies inside a server room using a purposely designed graph neural network architecture that is compliant with the fundamental time-reversal symmetry. Because of its well-behaved and generalizable state-action representations, the model enables sample-efficient and robust latent space offline policy learning using limited real-world operational data. Our framework has been successfully deployed and verified in a large-scale production DC for closed-loop control of its air-cooling units (ACUs). We conducted a total of 2000 hours of short and long-term experiments in the production DC environment. The results show that our method achieves 14~21% energy savings in the DC cooling system, without any violation of the safety or operational constraints. Our results have demonstrated the significant potential of offline RL in solving a broad range of data-limited, safety-critical real-world industrial control problems.

Figures (12)

• Figure 1: Illustration of the DC floor-level cooling system and temperature fields under different server loads.
• Figure 2: Illustration of the physics-informed offline RL framework for energy-efficient DC cooling control.
• Figure 3: Comparisons of key system metrics and the controllable actions of our method and the PID controller over 2-day testing periods in Server Room B. Figures on the left show results from the PID-controlled period (May 13-15, 2024), and figures on the right are the results controlled by our method (June 29 - July 1, 2024).
• Figure 4: Results of the 14-day long-term experiments in Server Room B. a, ACLF values under different total server loads. b, c, Temperature distribution of the directly influenced hot and cold aisles.
• Figure 6: Comparative evaluation of our method against baseline methods on our real-world testbed.