Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GPU Under Pressure: Estimating Application's Stress via Telemetry and Performance Counters

Giuseppe Esposito, Juan-David Guerrero-Balaguera, Josie Esteban Rodriguez Condia, Matteo Sonza Reorda, Marco Barbiero, Rossella Fortuna

TL;DR

The paper tackles predicting GPU reliability under sustained workloads by estimating workload stress through a fusion of online telemetry and hardware performance counters. It proposes a three-stage methodology: high-level performance profiling with roofline analysis to select representative performance counters; fine-grain measurement via CUPTI; and telemetry monitoring with NVML, focusing on $E$, Temperature, and Clock Frequency, as well as thermal metrics $t_r$ and $T_{\infty}$. Experiments with ten applications spanning GPU-Burn, CNNs, and Rodinia show distinct stress profiles, with GPU-Burn reaching near-peak throughput ($6450$ GFLOP/s) and a large DVFS swing ($\Delta CF$ up to $740$ MHz), while CNNs and Rodinia induce lower $S_{act}$ and $AII$ but still meaningful thermal impact. The results demonstrate that jointly analyzing telemetry and PCs yields reliable indicators of aging-related risk and can guide stress testing and workload deployment in data centers.

Abstract

Graphics Processing Units (GPUs) are specialized accelerators in data centers and high-performance computing (HPC) systems, enabling the fast execution of compute-intensive applications, such as Convolutional Neural Networks (CNNs). However, sustained workloads can impose significant stress on GPU components, raising reliability concerns due to potential faults that corrupt the intermediate application computations, leading to incorrect results. Estimating the stress induced by an application is thus crucial to predict reliability (with\,special\,emphasis\,on\,aging\,effects). In this work, we combine online telemetry parameters and hardware performance counters to assess GPU stress induced by different applications. The experimental results indicate the stress induced by a parallel workload can be estimated by combining telemetry data and Performance Counters that reveal the efficiency in the resource usage of the target workload. For this purpose the selected performance counters focus on measuring the i) throughput, ii) amount of issued instructions and iii) stall events.

GPU Under Pressure: Estimating Application's Stress via Telemetry and Performance Counters

TL;DR

The paper tackles predicting GPU reliability under sustained workloads by estimating workload stress through a fusion of online telemetry and hardware performance counters. It proposes a three-stage methodology: high-level performance profiling with roofline analysis to select representative performance counters; fine-grain measurement via CUPTI; and telemetry monitoring with NVML, focusing on , Temperature, and Clock Frequency, as well as thermal metrics and . Experiments with ten applications spanning GPU-Burn, CNNs, and Rodinia show distinct stress profiles, with GPU-Burn reaching near-peak throughput ( GFLOP/s) and a large DVFS swing ( up to MHz), while CNNs and Rodinia induce lower and but still meaningful thermal impact. The results demonstrate that jointly analyzing telemetry and PCs yields reliable indicators of aging-related risk and can guide stress testing and workload deployment in data centers.

Abstract

Graphics Processing Units (GPUs) are specialized accelerators in data centers and high-performance computing (HPC) systems, enabling the fast execution of compute-intensive applications, such as Convolutional Neural Networks (CNNs). However, sustained workloads can impose significant stress on GPU components, raising reliability concerns due to potential faults that corrupt the intermediate application computations, leading to incorrect results. Estimating the stress induced by an application is thus crucial to predict reliability (with\,special\,emphasis\,on\,aging\,effects). In this work, we combine online telemetry parameters and hardware performance counters to assess GPU stress induced by different applications. The experimental results indicate the stress induced by a parallel workload can be estimated by combining telemetry data and Performance Counters that reveal the efficiency in the resource usage of the target workload. For this purpose the selected performance counters focus on measuring the i) throughput, ii) amount of issued instructions and iii) stall events.

Paper Structure

This paper contains 12 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Stress analysis for GPU applications
  3. High-level performance profiling
  4. Fine-grain Performance Measurement
  5. Telemetry monitoring
  6. Experimental setup
  7. Experimental results
  8. Roofline analysis
  9. Performance stress indicators
  10. Performance stress indicators
  11. Performance Metrics impact on Thermal and Dynamic Stress
  12. Conclusion and future works

Figures (3)

  • Figure 1: Two different workloads can have varying degrees of parallelism, which indicates a different number of threads executed simultaneously. A higher number of parallel threads requires greater resource usage, including computational resources, memory, and internal activity. As resource usage increases, performance counters that track internal activity and in-field telemetry sensors register greater variations in their recorded values. Additionally, the sensors embedded in the device show a larger variation in their readings.
  • Figure 2: Roofline model analysis for the evaluated benchmarks.
  • Figure 3: Radar chart showing the telemetry and PCs data for the evaluated workloads (i.e., GPU-burndefour2013gpuburn, CNNs marcel2010torchvision and Rodinia benchmarks che2009rodinia.