Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Host-Side Telemetry for Performance Diagnosis in Cloud and HPC GPU Infrastructure

Erfan Darzi, Aldo Pareja, Shreeanant Bharadwaj

TL;DR

This work tackles GPU tail-latency in cloud and HPC environments by introducing an eBPF-based host-side telemetry system that unifies host signals with GPU-internal events to enable real-time root-cause analysis. The method combines a telemetry collection agent and a correlation engine, leveraging a four-layer pipeline, frequent sampling, spike scoring, and lagged cross-correlation to identify and rank interference sources. Key contributions include a unified telemetry framework, a correlation engine achieving $81\%$–$88\%$ diagnostic accuracy, $5$-second detection latency, and <$2\%$ CPU overhead, demonstrated on distributed learning workloads with root causes such as NIC contention, PCIe pressure, and CPU interference. The results show practical applicability for operational debugging of multi-tenant GPU infrastructure without requiring cluster-wide instrumentation, with natural extension to multi-node clusters for cluster-wide RCA.

Abstract

Diagnosing GPU tail latency spikes in cloud and HPC infrastructure is critical for maintaining performance predictability and resource utilization, yet existing monitoring tools lack the granularity for root cause analysis in shared computing environments. We introduce an eBPF-based telemetry system that provides unified host-side monitoring of GPU workloads, correlating eBPF-derived host metrics with GPU-internal events for holistic system observability. The system achieves 81--88\% diagnostic accuracy, detects spikes within 5 seconds, and completes root cause analysis in 6--8 seconds, operating with 1.21\% CPU overhead at 100Hz sampling. Evaluated on distributed learning workloads, the system identifies root causes including NIC contention, PCIe pressure, and CPU interference, enabling operational debugging for multi-tenant GPU infrastructure without requiring cluster-wide instrumentation.

Host-Side Telemetry for Performance Diagnosis in Cloud and HPC GPU Infrastructure

TL;DR

This work tackles GPU tail-latency in cloud and HPC environments by introducing an eBPF-based host-side telemetry system that unifies host signals with GPU-internal events to enable real-time root-cause analysis. The method combines a telemetry collection agent and a correlation engine, leveraging a four-layer pipeline, frequent sampling, spike scoring, and lagged cross-correlation to identify and rank interference sources. Key contributions include a unified telemetry framework, a correlation engine achieving – diagnostic accuracy, -second detection latency, and < CPU overhead, demonstrated on distributed learning workloads with root causes such as NIC contention, PCIe pressure, and CPU interference. The results show practical applicability for operational debugging of multi-tenant GPU infrastructure without requiring cluster-wide instrumentation, with natural extension to multi-node clusters for cluster-wide RCA.

Abstract

Diagnosing GPU tail latency spikes in cloud and HPC infrastructure is critical for maintaining performance predictability and resource utilization, yet existing monitoring tools lack the granularity for root cause analysis in shared computing environments. We introduce an eBPF-based telemetry system that provides unified host-side monitoring of GPU workloads, correlating eBPF-derived host metrics with GPU-internal events for holistic system observability. The system achieves 81--88\% diagnostic accuracy, detects spikes within 5 seconds, and completes root cause analysis in 6--8 seconds, operating with 1.21\% CPU overhead at 100Hz sampling. Evaluated on distributed learning workloads, the system identifies root causes including NIC contention, PCIe pressure, and CPU interference, enabling operational debugging for multi-tenant GPU infrastructure without requiring cluster-wide instrumentation.
Paper Structure (13 sections, 2 equations, 3 figures, 4 tables)

This paper contains 13 sections, 2 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Telemetry and Signals
  4. Diagnosis and Correlation
  5. Evaluation
  6. Workloads and Disturbances
  7. Baselines and Metrics
  8. Results
  9. Comparison with Prior Approaches
  10. Discussion and Limitations
  11. Extension to Multi-Node GPU Clusters
  12. Related Work
  13. Conclusion

Figures (3)

  • Figure 1: System architecture showing four-layer telemetry and correlation pipeline. Layer 1 collects multi-source signals via eBPF and NVML. Layer 2 performs temporal synchronization and 3$\sigma$ spike detection. Layer 3 executes lagged cross-correlation and confidence scoring. Layer 4 outputs ranked root causes.
  • Figure 2: Performance of the eBPF telemetry system. (a) The trade-off between CPU overhead and detection latency. (b) Box plot of classification accuracy across 17 runs for each disturbance type.
  • Figure 3: Timeline of signals during a NIC burst. The plot shows the correlation between NCCL latency and NET_RX softirqs, leading to a diagnosis.