Methodology for GPU Frequency Switching Latency Measurement

TL;DR

The paper develops a robust methodology to quantify GPU frequency switching latency, addressing the gap between CPU DVFS paradigms and accelerator behavior. By combining a per-core microbenchmark, synchronized timing, and adaptive statistical filtering, it reveals substantial cross-device variability in latency across Nvidia GPUs. The CUDA-based tool implements the methodology and provides actionable data to design energy-aware runtime systems that optimize or avoid costly frequency transitions. Overall, the work advances predictable energy savings for accelerators in exascale and post-exascale systems by characterizing real-world switching overheads.

Abstract

The development of exascale and post-exascale HPC and AI systems integrates thousands of CPUs and specialized accelerators, making energy optimization critical as power costs rival hardware expenses. To reduce consumption, frequency and voltage scaling techniques are widely used, but their effectiveness depends on adapting to application demands in real-time. However, frequency scaling incurs a switching latency, impacting the responsiveness of dynamic tuning approaches. We propose a methodology to systematically evaluate the frequency switching latency of accelerators, with an implementation for CUDA. Our approach employs an artificial iterative workload designed for precise runtime measurements at different frequencies. The methodology consists of three phases: (1) measuring workload execution time across target frequencies, (2) determining switching latency by tracking the transition from an initial to a target frequency, and (3) filtering out outliers caused by external factors such as CUDA driver management or CPU interruptions. A robust statistical system ensures accuracy while minimizing execution time. We demonstrate this methodology on three Nvidia GPUs - GH200, A100, and RTX Quadro 6000 - revealing significant variations in switching latency. These findings are crucial for designing energy-efficient runtime systems, helping determine optimal frequency change rates and avoiding transitions with excessive overhead.

Figures (9)

• Figure 1: Visualization of the Intel Skylake SP CPU skylake-ucf-lat behavior when processing the frequency change request.
• Figure 2: Visualization of CPU to ACC communication while issuing the ACC frequency change request. The dashed line shows the frequency change.
• Figure 3: Heatmaps of minimum or maximum switching latencies [ms] for a selected set of frequencies [MHz]. The green color identifies the smallest values in the heatmap, and the red color identifies the highest values. Initial frequencies in rows, target frequencies in columns.
• Figure 4: Switching latency distribution. Comparing distribution when streaming multiprocessor frequency increasing (left) to the frequency decreasing (right).
• Figure 5: Scatter plot of measured switching latencies from 1770 to 1260 MHz on GH200.