Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Trivance: Latency-Optimal AllReduce by Shortcutting Multiport Networks

Anton Juerss, Vamsi Addanki, Stefan Schmid

TL;DR

Trivance is a novel AllReduce algorithm that completes within $\log_3 n$ steps, while reducing congestion compared to Bruck's algorithm by a factor of three and preserving bandwidth-optimality, and extends naturally to multidimensional torus networks.

Abstract

AllReduce is a fundamental collective operation in distributed computing and a key performance bottleneck for large-scale training and inference. Its completion time is determined by the number of communication steps, which dominates latency-sensitive workloads, and the communication distance affecting both latency- and bandwidth-bound regimes. Direct-connect topologies, such as torus networks used in Google's TPUv4, are particularly prone to large communication distances due to limited bisection bandwidth. Latency-optimal algorithms such as Bruck's complete AllReduce in $\log_3 n$ steps on a bidirectional ring, but incur large communication distances that result in substantial congestion. In contrast, recent approaches such as Swing reduce communication distance and congestion, but are inherently required to perform $\log_2 n$ steps to complete AllReduce, sacrificing latency-optimality. In this paper, we present Trivance, a novel AllReduce algorithm that completes within $\log_3 n$ steps, while reducing congestion compared to Bruck's algorithm by a factor of three and preserving bandwidth-optimality. Trivance exploits both transmission ports of a bidirectional ring within each step to triple the communication distance along both directions simultaneously. Furthermore, by performing joint reductions, Trivance improves both the number of steps and network congestion. We further show that Trivance extends naturally to multidimensional torus networks, retaining its latency advantage while achieving performance comparable to bandwidth-optimal algorithms for large messages. Our empirical evaluation shows that Trivance improves state-of-the-art approaches by 5-30% for message sizes up to 8\,MiB, in high-bandwidth settings up to 32MiB and for 3D tori up to 128MiB. Throughout the evaluation, Trivance remains the best-performing latency-optimal algorithm.

Trivance: Latency-Optimal AllReduce by Shortcutting Multiport Networks

TL;DR

Trivance is a novel AllReduce algorithm that completes within steps, while reducing congestion compared to Bruck's algorithm by a factor of three and preserving bandwidth-optimality, and extends naturally to multidimensional torus networks.

Abstract

AllReduce is a fundamental collective operation in distributed computing and a key performance bottleneck for large-scale training and inference. Its completion time is determined by the number of communication steps, which dominates latency-sensitive workloads, and the communication distance affecting both latency- and bandwidth-bound regimes. Direct-connect topologies, such as torus networks used in Google's TPUv4, are particularly prone to large communication distances due to limited bisection bandwidth. Latency-optimal algorithms such as Bruck's complete AllReduce in steps on a bidirectional ring, but incur large communication distances that result in substantial congestion. In contrast, recent approaches such as Swing reduce communication distance and congestion, but are inherently required to perform steps to complete AllReduce, sacrificing latency-optimality. In this paper, we present Trivance, a novel AllReduce algorithm that completes within steps, while reducing congestion compared to Bruck's algorithm by a factor of three and preserving bandwidth-optimality. Trivance exploits both transmission ports of a bidirectional ring within each step to triple the communication distance along both directions simultaneously. Furthermore, by performing joint reductions, Trivance improves both the number of steps and network congestion. We further show that Trivance extends naturally to multidimensional torus networks, retaining its latency advantage while achieving performance comparable to bandwidth-optimal algorithms for large messages. Our empirical evaluation shows that Trivance improves state-of-the-art approaches by 5-30% for message sizes up to 8\,MiB, in high-bandwidth settings up to 32MiB and for 3D tori up to 128MiB. Throughout the evaluation, Trivance remains the best-performing latency-optimal algorithm.
Paper Structure (38 sections, 3 theorems, 10 equations, 11 figures, 2 tables, 1 algorithm)

This paper contains 38 sections, 3 theorems, 10 equations, 11 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Model and Preliminaries
  3. Latency, Bandwidth & Congestion
  4. AllReduce as a Collective Operation
  5. Optimality for $D$-dimensional torus networks for Latency and Bandwidth
  6. State-of-the-art Algorithms
  7. Hamiltonian Rings and Bucket:
  8. Recursive Doubling:
  9. Swing:
  10. Bruck:
  11. Bidirectional Design of Swing, Recursive Doubling and Bucket:
  12. Motivation
  13. Problems of Existing Approaches
  14. Exploration of Design Space
  15. Trivance
  16. ...and 23 more sections

Key Result

lemma 1

The Trivance algorithm achieves bandwidth-optimality by distributing $m \cdot (n - 1)$ unique bytes in $\log_3 n$ steps per Reduce-Scatter or AllGather phase.

Figures (11)

  • Figure 1: Compared to Recursive Doubling, Trivance completes in $\log_3 n$ steps by leveraging both ports for joint reductions and tripled communication distance per step. Congestion is reduced by factor three compared to Bruck.
  • Figure 2: Progression of received blocks at node $r$ from step $k$ to step $k+1$, acquiring the blocks of nodes $p$ and $q$ along with their neighbors, tripling the known data.
  • Figure 3: Progression of received blocks for each node on a ring of size $n = 9$.
  • Figure 4: Trivance AllReduce for networks of size 7 (left) and 9 (right).
  • Figure 5: First two steps of Trivance AllReduce for a square torus of size $n=81$. Green arrows show the original collective; purple the mirrored collective.
  • ...and 6 more figures

Theorems & Definitions (5)

  • lemma 1: Bandwidth-optimality
  • lemma 2: Block propagation
  • theorem 1: Latency-optimality
  • proof
  • proof : Proof by contradiction