Coordinated Cooling and Compute Management for AI Datacenters

TL;DR

This paper tackles the energy efficiency and thermal sustainability of AI datacenters hosting LLM inference by introducing a hierarchical compute–thermal control framework that co-optimizes GPU parallelism, DVFS, and cooling settings. It combines an LSTM-based workload forecaster with a DistilBERT-based job classifier to drive MILP- and MPC-based optimization across TP selection, workload provisioning, and cooling control. Using real Azure inference traces and detailed GPU profiling, the approach achieves significant reductions in computing energy (~24%) and cooling energy (~31%), while maintaining end-to-end latency within SLOs and reducing GPU temperatures. The work demonstrates the importance and practicality of integrated management across compute and cooling for scalable, sustainable AI serving infrastructures and points to future extensions with more advanced cooling modalities.

Abstract

The AI datacenters are currently being deployed on a large scale to support the training and deployment of power-intensive large-language models (LLMs). Extensive amount of computation and cooling required in datacenters increase concerns about the energy use and carbon emissions of AI datacenters. Although current state-of-the-art has examined the energy efficiency of LLM inference, most prior research focused on optimizing compute-side scheduling without considering thermal objectives or constraints. Since GPU-intensive inference generates substantial heat that can degrade datacenter performance, ignoring thermal effects can increase total energy consumption and reduce the efficiency of LLM serving. To fill this gap, we profile the characteristics of GPU servers under varying cooling and AI jobs, and develop a joint cooling and computing modeling approach for AI datacenters. Built upon such workload and thermal dynamics models, a novel hierarchical control framework is proposed to co-optimize computing and thermal management by identifying the optimal GPU parallelism, frequency (DVFS), and cooling control knobs. Using real Azure inference traces and detailed GPU profiling, our model balances serving latency and thermal constraints in AI datacenters while significantly improving AI datacenters' energy efficiency.

Figures (8)

• Figure 1: Schematic for proposed hierarchical control of joint cooling–compute for AI datacenters.
• Figure 2: Schematic of airflow in the hot and cold zone, with cooling from Rack Cooling Units (RCU).
• Figure 3: Simulation results of the hierarchical control framework integrating workload utilization, cluster-level provisioning, MILP-based frequency tuning, computing power consumption, and cooling regulation. The bottom panels show the comparison between PID and MPC controllers in the cooling layer, illustrating the control efforts required by supply air temperature and airflow rate from both controllers.
• Figure 4: High-resolution one hour windows in two operating regimes: high-traffic (left, panels) and low-load (right, panels). Top rows show workload utilization, LP-based provisioning, and MILP-based GPU frequency optimization; bottom rows show MPC-regulated thermals: return temperature, inlet temperature, airflow rate, and supply temperature. This side-by-side view highlights control behavior and thermal--power responses under peak versus trough demand.
• Figure 5: Performance on the computing side versus the cooling side with the proposed control framework versus a baseline on the LLaMA-2-7B inference on a real 8$\times$Tesla (16 GB) GPU server. The plots collectively depict (a) end-to-end inference latency (QPS), (b) GPU temperature, (c) average per-GPU power consumption, and (d) cooling power.