Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ethereal: Divide and Conquer Network Load Balancing in Large-Scale Distributed Training

Vamsi Addanki, Prateesh Goyal, Ilias Marinos, Stefan Schmid

TL;DR

This paper challenges the prevailing assumption that packet spraying is required for high-performance large-scale distributed training. It introduces Ethereal, a simple singlepath transport augmented by application-layer flow splitting, which, under collective communication patterns, achieves near-optimal congestion balance comparable to packet spraying while maintaining per-flow context for effective failure handling. A formal equivalence result shows that small-scale flow splitting can match the optimal load distribution of multipath spraying, and the Ethereal design—flow interception, divide-and-conquer path assignment, and RoCE-based loss recovery—yields substantial reductions in collective completion times in simulations and supports robust operation under failures. The work provides a practical, NIC-friendly transport approach tailored to distributed training workloads, with strong empirical evidence from Astra-Sim and Chakra indicating significant performance gains and deployment viability.

Abstract

Large-scale distributed training in production datacenters constitutes a challenging workload bottlenecked by network communication. In response, both major industry players (e.g., Ultra Ethernet Consortium) and parts of academia have surprisingly, and almost unanimously, agreed that packet spraying is \emph{necessary} to improve the performance of large-scale distributed training workloads. In this paper, we challenge this prevailing belief and pose the question: \emph{How close can singlepath transport come to matching the performance of packet spraying?} We demonstrate that singlepath transport (from a NIC's perspective) is sufficient and can perform nearly as well as ideal packet spraying, particularly in the context of distributed training in CLOS-based topologies. Our assertion is based on four key observations about workloads driven by collective communication patterns: \emph{(i)} flow sizes are known upon arrival, \emph{(ii)} flow sizes are equal within each step of a collective, \emph{(iii)} the completion time of a collective is more critical than individual flow completion times, and \emph{(iv)} flows can be \emph{split} upon arrival to control load balancing directly from the application layer. We present Ethereal, a simple distributed load balancing algorithm that opportunistically splits flows and assigns paths to each flow in a transparent manner, requiring little to no changes to existing RDMA NICs. Our evaluation, spanning a wide range of collective communication algorithms and GPT models using Astra-Sim, shows that Ethereal significantly reduces the completion times by up to $30\%$ compared to packet spraying and by up to $40\%$ compared to REPS, even under link failures. This paper offers an alternative perspective for developing next-generation transport protocols tailored to large-scale distributed training.

Ethereal: Divide and Conquer Network Load Balancing in Large-Scale Distributed Training

TL;DR

This paper challenges the prevailing assumption that packet spraying is required for high-performance large-scale distributed training. It introduces Ethereal, a simple singlepath transport augmented by application-layer flow splitting, which, under collective communication patterns, achieves near-optimal congestion balance comparable to packet spraying while maintaining per-flow context for effective failure handling. A formal equivalence result shows that small-scale flow splitting can match the optimal load distribution of multipath spraying, and the Ethereal design—flow interception, divide-and-conquer path assignment, and RoCE-based loss recovery—yields substantial reductions in collective completion times in simulations and supports robust operation under failures. The work provides a practical, NIC-friendly transport approach tailored to distributed training workloads, with strong empirical evidence from Astra-Sim and Chakra indicating significant performance gains and deployment viability.

Abstract

Large-scale distributed training in production datacenters constitutes a challenging workload bottlenecked by network communication. In response, both major industry players (e.g., Ultra Ethernet Consortium) and parts of academia have surprisingly, and almost unanimously, agreed that packet spraying is \emph{necessary} to improve the performance of large-scale distributed training workloads. In this paper, we challenge this prevailing belief and pose the question: \emph{How close can singlepath transport come to matching the performance of packet spraying?} We demonstrate that singlepath transport (from a NIC's perspective) is sufficient and can perform nearly as well as ideal packet spraying, particularly in the context of distributed training in CLOS-based topologies. Our assertion is based on four key observations about workloads driven by collective communication patterns: \emph{(i)} flow sizes are known upon arrival, \emph{(ii)} flow sizes are equal within each step of a collective, \emph{(iii)} the completion time of a collective is more critical than individual flow completion times, and \emph{(iv)} flows can be \emph{split} upon arrival to control load balancing directly from the application layer. We present Ethereal, a simple distributed load balancing algorithm that opportunistically splits flows and assigns paths to each flow in a transparent manner, requiring little to no changes to existing RDMA NICs. Our evaluation, spanning a wide range of collective communication algorithms and GPT models using Astra-Sim, shows that Ethereal significantly reduces the completion times by up to compared to packet spraying and by up to compared to REPS, even under link failures. This paper offers an alternative perspective for developing next-generation transport protocols tailored to large-scale distributed training.
Paper Structure (14 sections, 1 theorem, 8 figures, 1 table, 1 algorithm)

This paper contains 14 sections, 1 theorem, 8 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation
  3. Design Choices for Load Balancing
  4. Limitations of Existing Approaches
  5. Making a Case for Singlepath Transport
  6. Ethereal Transport for AI
  7. Flow Interceptor
  8. Divide and Conquer Load Balancing
  9. Loss Recovery and Handling Failures
  10. Evaluation
  11. Setup
  12. Results
  13. Related Work
  14. Conclusion

Key Result

Theorem 1

Given a leaf-spine topology, with $\ell$ leaves, $s$ spines and $k$ server nodes, a set of demands $\mathcal{M}=\{ f_{i}\times n_{i,j} \mid f_{i}, n_{i,j}\in \mathbb{N}, i\in [1,k], j\in[1,\ell] \}$, where $f_{i}$ is the flow size and $n_{i,j}$ is the number of flows that a server node $i$ has towar

Figures (8)

  • Figure 1: In contrast to traditional datacenter workloads, distributed training workloads exhibit certain properties in terms of flow sizes, the number of concurrent flows, and arrival times that allow singlepath transport to achieve nearly the same performance as an optimal multipath transport. The problem essentially boils down to assigning paths to each flow in order to minimize congestion.
  • Figure 2: Packet spraying struggles with congestion control under asymmetries, whereas, singlepath adaptive load balancing struggles with path flapping even in a symmetric topology.
  • Figure 3: Leaf spine topology with $256$ GPUs: Ethereal outperforms in completion times for all-to-all and recursive doubling without significantly increasing the number of required queue pairs at the NIC.
  • Figure 4: Fat-tree topology with $512$ GPUs: Even in a hierarchical topology, Ethereal significantly improves the completion times for allReduce with all-to-all and recursive doubling, while performing similarly to other approaches under ring and double binary tree.
  • Figure 5: Leaf spine topology GPT-2 (one iteration)
  • ...and 3 more figures

Theorems & Definitions (1)

  • Theorem 1: Equivalence