Reducing Compute Waste in LLMs through Kernel-Level DVFS

TL;DR

This work tackles the high operational energy footprint of large language models by reframing DVFS from a coarse, iteration-level tactic to a fine-grained kernel-level control, paired with a compute-waste objective that preserves throughput. By applying kernel-level DVFS to a GPT-3-scale case study, the authors demonstrate up to $14.6\%$ energy savings with only $0.6\%$ slowdown, outperforming pass-level approaches. The study validates that per-kernel frequency configurations translate across data- and tensor-parallelism, and shows substantial benefits even under strict waste-reduction policies. These findings suggest that modest energy reductions at scale can yield large absolute savings for both training and potentially inference in real-world AI workloads, with manageable impacts on performance and practical DVFS implementation considerations.

Abstract

The rapid growth of AI has fueled the expansion of accelerator- or GPU-based data centers. However, the rising operational energy consumption has emerged as a critical bottleneck and a major sustainability concern. Dynamic Voltage and Frequency Scaling (DVFS) is a well-known technique used to reduce energy consumption, and thus improve energy-efficiency, since it requires little effort and works with existing hardware. Reducing the energy consumption of training and inference of Large Language Models (LLMs) through DVFS or power capping is feasible: related work has shown energy savings can be significant, but at the cost of significant slowdowns. In this work, we focus on reducing waste in LLM operations: i.e., reducing energy consumption without losing performance. We propose a fine-grained, kernel-level, DVFS approach that explores new frequency configurations, and prove these save more energy than previous, pass- or iteration-level solutions. For example, for a GPT-3 training run, a pass-level approach could reduce energy consumption by 2% (without losing performance), while our kernel-level approach saves as much as 14.6% (with a 0.6% slowdown). We further investigate the effect of data and tensor parallelism, and show our discovered clock frequencies translate well for both. We conclude that kernel-level DVFS is a suitable technique to reduce waste in LLM operations, providing significant energy savings with negligible slow-down.

Figures (8)

• Figure 1: An overview of one LLM training iteration.
• Figure 2: Desirability of configurations according to the EDP and waste optimization goals. Darker colors indicate better scores.
• Figure 3: Forward pass. Depicts the percentage of time and energy gained or lost when the reducing the memory clock and core clock from the (auto, auto) baseline (at the origin).
• Figure 4: Backward pass. Depicts the percentage of time and energy gained or lost when the reducing the memory clock and core clock from the (auto, auto) baseline (at the origin).
• Figure 5: Absolute time and energy for all kernels. Dots indicate the auto configuration, while the error bars indicate the minimum and maximum value achieved by any clock configuration.