Kareus: Joint Reduction of Dynamic and Static Energy in Large Model Training

TL;DR

Kareus tackles the energy bottleneck in large-model training by jointly optimizing dynamic energy (via GPU frequency) and static energy (via kernel scheduling and resource underutilization). It introduces a partitioned overlap execution model to decompose the joint time-energy optimization into tractable subproblems and uses a multi-objective Bayesian optimization framework to construct the time-energy frontier for each partition before composing them into an iteration-level frontier. By integrating with Perseus and Megatron-LM, Kareus demonstrates up to 28.3% energy reduction at the same time and up to 27.5% time reduction at the same energy across diverse workloads and scales, validated on real hardware and large-scale emulation. The work shows that joint execution scheduling and power control can yield substantial practical gains, suggesting energy-aware scheduling as a core design principle for scalable ML systems.

Abstract

The computing demand of AI is growing at an unprecedented rate, but energy supply is not keeping pace. As a result, energy has become an expensive, contended resource that requires explicit management and optimization. Although recent works have made significant progress in large model training optimization, they focus only on a single aspect of energy consumption: dynamic or static energy. We find that fine-grained kernel scheduling and frequency scaling jointly and interdependently impact both dynamic and static energy consumption. Based on this finding, we design Kareus, a training system that pushes the time--energy tradeoff frontier by optimizing both aspects. Kareus decomposes the intractable joint optimization problem into local, partition-based subproblems. It then uses a multi-pass multi-objective optimization algorithm to find execution schedules that push the time--energy tradeoff frontier. Compared to the state of the art, Kareus reduces training energy by up to 28.3% at the same training time, or reduces training time by up to 27.5% at the same energy consumption.

Figures (12)

• Figure 1: Existing training systems running the 1F1B pipeline schedule megatronlm-sc21. Redder colors indicate higher GPU frequency and power draw. (a) The baseline Megatron-LM megatronlm-sc21 does not perform any frequency scaling, whereas (b) Perseus perseus-sosp24 scales GPU frequencies of non-critical forward and backward computations to reduce energy consumption while maintaining the same latency.
• Figure 2: Transformer transformer-neurips17 Attention layer with tensor parallelism run with different execution models. (a) The sequential kernel execution model only runs one kernel at a time strictly following data dependencies. (b) Nanobatching domino-arxiv24deepseek-v3-arxiv24nanoflow-osdi25 splits a pipeline microbatch into two nanobatches (illustrated with different colors) and staggers their execution, creating opportunities to overlap communication and computation.
• Figure 3: The time and total energy consumption of execution schedules for one Transformer Attention layer forward pass with varying SM allocation, communication launch timing, and GPU frequency. (a)--(c) show the effect of allocating different numbers of SMs to the communication kernel at 1,410 MHz, with (b) being the energy-optimal schedule. (d) is the same as (b), except that communication was launched earlier together with Norm. Finally, (e) and (f) run at a lower GPU frequency of 1,100 MHz. (e) is the same as (d) other than frequency, whereas (f) is the energy-optimal schedule at 1,100 MHz, which is different from the schedule in (b).
• Figure 4: Kareus execution and optimization overview. The first row shows one training iteration with the 1F1B pipeline schedule. Inside each microbatch, partitions are sequentially executed; the second row illustrates this with the Transformer forward pass with tensor parallelism. The third row shows the execution schedule of the Attention--AllReduce partition. Each partition's execution schedule is chosen from the partition-level time--energy frontiers (fourth row) characterized by Kareus's optimization algorithm.
• Figure 5: HVI.

Theorems & Definitions (3)

• Theorem 1
• Theorem 1: Energy Efficiency of Constant Frequency
• proof