Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Easy and Realistic Network Infrastructure Testing for Large-scale Machine Learning

Jinsun Yoo, ChonLam Lao, Lianjie Cao, Bob Lantz, Minlan Yu, Tushar Krishna, Puneet Sharma

TL;DR

Large-scale ML training is often bottlenecked by network performance across distributed GPUs. Genie provides CPU-based traffic generation that emulates GPU-to-GPU communication and leverages ASTRA-sim to model the interaction between network behavior and the ML workload, with Chakra graphs representing workload structure. The framework enables realistic testing of network configurations, failures, and congestion control on real networks without requiring GPUs by injecting traffic and feeding workload dynamics to the simulator. This GPU-free, network-aware testing approach offers a scalable path for validating network designs and optimizations in production-like ML clusters.

Abstract

This paper lays the foundation for Genie, a testing framework that captures the impact of real hardware network behavior on ML workload performance, without requiring expensive GPUs. Genie uses CPU-initiated traffic over a hardware testbed to emulate GPU to GPU communication, and adapts the ASTRA-sim simulator to model interaction between the network and the ML workload.

Towards Easy and Realistic Network Infrastructure Testing for Large-scale Machine Learning

TL;DR

Large-scale ML training is often bottlenecked by network performance across distributed GPUs. Genie provides CPU-based traffic generation that emulates GPU-to-GPU communication and leverages ASTRA-sim to model the interaction between network behavior and the ML workload, with Chakra graphs representing workload structure. The framework enables realistic testing of network configurations, failures, and congestion control on real networks without requiring GPUs by injecting traffic and feeding workload dynamics to the simulator. This GPU-free, network-aware testing approach offers a scalable path for validating network designs and optimizations in production-like ML clusters.

Abstract

This paper lays the foundation for Genie, a testing framework that captures the impact of real hardware network behavior on ML workload performance, without requiring expensive GPUs. Genie uses CPU-initiated traffic over a hardware testbed to emulate GPU to GPU communication, and adapts the ASTRA-sim simulator to model interaction between the network and the ML workload.
Paper Structure (10 sections, 3 figures)

This paper contains 10 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Background
  3. ML Training Process Behavior
  4. Communication in ML Training
  5. Representing ML Workload
  6. Motivation
  7. Studying the Impact of Network on AI/ML Workload
  8. Easily Verifying Network Integrity
  9. G ENIE Design
  10. Conclusion and Future Work

Figures (3)

  • Figure 1: Bandwidth for several AllReduce runs against simulation. We inject an anomaly which the simulation does not anticipate.
  • Figure 2: A typical deployment of training processes across multiple nodes. The blue arrow depicts the communication between nodes.
  • Figure 3: Overall Architecture of G ENIE. G ENIE can work on a cluster with only CPU nodes connected via the network infrastructure.