Offline Reinforcement-Learning-Based Power Control for Application-Agnostic Energy Efficiency

TL;DR

This work tackles the challenge of reducing energy consumption in HPC nodes without harming application performance by training an application- and hardware-agnostic power controller offline. It employs Conservative Q-Learning on a precollected dataset of system states, actions, and rewards, using a state vector that includes progress, power, and hardware counters, and a discrete set of CPU power caps via RAPL. The method demonstrates about $20 ext{-} ext{percent}$ energy savings with roughly $7 ext{–}7.5 ext{ percent}$ performance degradation on unseen, phase-varied benchmarks, outperforming several baselines on the energy-performance Pareto frontier and showing robust, low-overhead operation. This approach has practical potential for datacenters and HPC clusters, enabling energy-efficient runtime control without live-system risk or costly surrogate modeling, with future extensions to GPUs and larger-scale deployments.

Abstract

Energy efficiency has become an integral aspect of modern computing infrastructure design, impacting the performance, cost, scalability, and durability of production systems. The incorporation of power actuation and sensing capabilities in CPU designs is indicative of this, enabling the deployment of system software that can actively monitor and adjust energy consumption and performance at runtime. While reinforcement learning (RL) would seem ideal for the design of such energy efficiency control systems, online training presents challenges ranging from the lack of proper models for setting up an adequate simulated environment, to perturbation (noise) and reliability issues, if training is deployed on a live system. In this paper we discuss the use of offline reinforcement learning as an alternative approach for the design of an autonomous CPU power controller, with the goal of improving the energy efficiency of parallel applications at runtime without unduly impacting their performance. Offline RL sidesteps the issues incurred by online RL training by leveraging a dataset of state transitions collected from arbitrary policies prior to training. Our methodology applies offline RL to a gray-box approach to energy efficiency, combining online application-agnostic performance data (e.g., heartbeats) and hardware performance counters to ensure that the scientific objectives are met with limited performance degradation. Evaluating our method on a variety of compute-bound and memory-bound benchmarks and controlling power on a live system through Intel's Running Average Power Limit, we demonstrate that such an offline-trained agent can substantially reduce energy consumption at a tolerable performance degradation cost.

Figures (5)

• Figure 1: Power vs instantaneous performance (progress) for the stream-full benchmark on an Intel Cascadelake node. Progress, defined in Eq. \ref{['eq:progress-calculation']}, is a proxy for the instantaneous performance of an application toward its scientific objective.
• Figure 2: Flow diagram for offline reinforcement learning: data is collected using an arbitrary policy to train the agent without system access, and the trained agent is later deployed for evaluation. Note that the lines in the diagram are not closed to form a loop, indicating offline training and online testing.
• Figure 3: Execution time vs. energy for twelve benchmarks. Six applications were unseen during training (Table \ref{['tab:repeatability']}). Yellow-red gradients show fixed $PCAP$ values; the green dot shows the proposed controller. Each point is averaged over five runs.
• Figure 4: Mean and standard deviation of $ED^2P$ vs. execution time for the experiments in Fig. /reffig:comparison. The proposed controller consistently maintains lower average $ED^2P$ across applications.
• Figure 5: Aggregate comparison between the proposed RL-based power-capping controller and four baselines: global PI, application-specific PI, DVFS, and the ondemand governor.