Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FAST: An Efficient Scheduler for All-to-All GPU Communication

Yiran Lei, Dongjoo Lee, Liangyu Zhao, Daniar Kurniawan, Chanmyeong Kim, Heetaek Jeong, Changsu Kim, Hyeonseong Choi, Liangcheng Yu, Arvind Krishnamurthy, Justine Sherry, Eriko Nurvitadhi

TL;DR

FAST tackles the problem of dynamic, skewed alltoallv on two-tier GPU fabrics by introducing a two-phase, online scheduler that first uses ultra-fast intra-server links to rebalance traffic and then applies Birkhoff decomposition to produce balanced, one-to-one inter-server transfers. The approach yields a pipelined end-to-end transfer that keeps the scale-out bottleneck saturated while hiding scale-up costs, achieving near-optimal completion times with negligible scheduling overhead. Empirically, FAST delivers state-of-the-art alltoallv throughput on NVIDIA and AMD clusters, reduces synthesis time from solver-based methods to microseconds (e.g., $25\mu$s for 32 GPUs, $221\mu$s for 64 GPUs), and provides substantial end-to-end gains in MoE workloads (up to $4.48\times$ in Megatron-LM). This work enables practical, real-time scheduling for dynamic MoE traffic, offering scalable performance improvements across modern two-tier GPU clusters.

Abstract

All-to-All(v) communication is a critical primitive in modern machine learning workloads, particularly mixture-of-experts (MoE) models. Unfortunately, efficient scheduling is challenging due to workload skew, heterogeneous two-tier fabrics, and incast congestion, compounded by the dynamic nature of MoE workloads, where traffic shifts every few hundred milliseconds. Existing schedulers are hardly scalable, incurring seconds to hours of synthesis time, making them impractical. We present FAST, an efficient All-to-All(v) scheduler. FAST addresses skew through intra-server rebalancing and enforces balanced, one-to-one scale-out transfers that avoid incast. Evaluated extensively on both NVIDIA H200 and AMD MI300X clusters, FAST consistently outperforms state-of-the-art solutions on skewed workloads while reducing synthesis time by orders of magnitude.

FAST: An Efficient Scheduler for All-to-All GPU Communication

TL;DR

FAST tackles the problem of dynamic, skewed alltoallv on two-tier GPU fabrics by introducing a two-phase, online scheduler that first uses ultra-fast intra-server links to rebalance traffic and then applies Birkhoff decomposition to produce balanced, one-to-one inter-server transfers. The approach yields a pipelined end-to-end transfer that keeps the scale-out bottleneck saturated while hiding scale-up costs, achieving near-optimal completion times with negligible scheduling overhead. Empirically, FAST delivers state-of-the-art alltoallv throughput on NVIDIA and AMD clusters, reduces synthesis time from solver-based methods to microseconds (e.g., s for 32 GPUs, s for 64 GPUs), and provides substantial end-to-end gains in MoE workloads (up to in Megatron-LM). This work enables practical, real-time scheduling for dynamic MoE traffic, offering scalable performance improvements across modern two-tier GPU clusters.

Abstract

All-to-All(v) communication is a critical primitive in modern machine learning workloads, particularly mixture-of-experts (MoE) models. Unfortunately, efficient scheduling is challenging due to workload skew, heterogeneous two-tier fabrics, and incast congestion, compounded by the dynamic nature of MoE workloads, where traffic shifts every few hundred milliseconds. Existing schedulers are hardly scalable, incurring seconds to hours of synthesis time, making them impractical. We present FAST, an efficient All-to-All(v) scheduler. FAST addresses skew through intra-server rebalancing and enforces balanced, one-to-one scale-out transfers that avoid incast. Evaluated extensively on both NVIDIA H200 and AMD MI300X clusters, FAST consistently outperforms state-of-the-art solutions on skewed workloads while reducing synthesis time by orders of magnitude.
Paper Structure (20 sections, 3 theorems, 2 equations, 18 figures)

This paper contains 20 sections, 3 theorems, 2 equations, 18 figures.

Table of Contents

  1. Introduction
  2. Motivation
  3. Design Overview
  4. Two Phase Scheduler
  5. Intra-server Scheduling: Balancing and Redistributing
  6. Inter-server Scheduling: Balanced, One-to-one Transfers
  7. End-to-End Transfer Pipeline
  8. Scheduler Properties
  9. Evaluation
  10. alltoallv Performance
  11. Performance Under Different Transfer Sizes
  12. Performance under Balanced All-to-All
  13. Performance under Different Skewness
  14. End-to-End Performance
  15. Scheduling Overhead
  16. ...and 5 more sections

Key Result

Theorem 1

The optimal transfer completion time $t_{optimal}$ is: .

Figures (18)

  • Figure 1: MoE models invoke alltoallv twice per MoE layer, making it a critical communication primitive.
  • Figure 2: All-to-All workload is skewed and dynamic when using Megatron-LM to pre-train a MoE model.
  • Figure 3: Workload skewness creates stragglers and underload NICs, leading to sub-optimal performance, which can be mitigated by rebalancing traffic.
  • Figure 4: Modern GPU clusters feature a two-tier fabric: a high-bandwidth server/rack–scale scale-up network (e.g., NVLink, Infinity Fabric) and a lower-bandwidth cross-server scale-out network (e.g., Ethernet, InfiniBand).
  • Figure 5: Birkhoff's decomposition of a 4-node All-to-All. Completion time is dictated by the largest sender ($N_0$ in blue), and the schedule is optimal since $N_0$ stays active in every stage while lighter nodes drop out early.
  • ...and 13 more figures

Theorems & Definitions (3)

  • Theorem 1
  • Theorem 2
  • Theorem 3