Efficient Direct-Connect Topologies for Collective Communications

TL;DR

This work tackles the challenge of designing efficient direct-connect topologies for collective communications by proposing an algorithmic toolchain that jointly optimizes topology and schedule. It introduces expansion techniques (Line Graph, Degree Expansion, Cartesian Product) to scale small, optimal base topologies into large, low-degree networks, and pairs them with a polynomial-time BFB schedule generation, validated on both real optical testbeds and large-scale simulations. A topology finder assembles Pareto-efficient options, and compilers map schedules to GPU and CPU runtimes, enabling substantial improvements in allreduce, allgather, and all-to-all performance for ML and HPC workloads. The results demonstrate order-of-magnitude reductions in communication time at scale and confirm the practicality of automated topology–schedule synthesis for reconfigurable direct-connect fabrics. The framework thus offers a scalable pathway to near-optimal, workload-aware direct-connect networks with broad impact on distributed training and HPC systems.

Abstract

We consider the problem of distilling efficient network topologies for collective communications. We provide an algorithmic framework for constructing direct-connect topologies optimized for the latency vs. bandwidth trade-off associated with the workload. Our approach synthesizes many different topologies and schedules for a given cluster size and degree and then identifies the appropriate topology and schedule for a given workload. Our algorithms start from small, optimal base topologies and associated communication schedules and use techniques that can be iteratively applied to derive much larger topologies and schedules. Additionally, we incorporate well-studied large-scale graph topologies into our algorithmic framework by producing efficient collective schedules for them using a novel polynomial-time algorithm. Our evaluation uses multiple testbeds and large-scale simulations to demonstrate significant performance benefits from our derived topologies and schedules.

Figures (22)

• Figure 1: The allgather schedule of complete bipartite graph $K_{2,2}$. Shard $S$ is divided into two half chunks $C_1$ and $C_2$. From $a$, at the 1st comm step, $a$ sends the entire shard $S$ to both $c$ and $d$. At the 2nd comm step, $c$ and $d$ send the two half chunks $C_1$ and $C_2$ respectively to $b$. Thus, every node receives the full shard from $a$. By applying similar broadcast from $c,b,d$ in parallel, we have a complete BW-optimal allgather schedule with $T_L\!=\!2\alpha,T_B\!=\!\frac{M}{B}\!\cdot\!\frac{3}{4}$.
• Figure 2: The complete bipartite topology $K_{2,2}$ with its line graph $L(K_{2,2})$. Figure (a) shows the base topology and broadcast paths from $a$ to $c,b,d$ in $A_{K_{2,2}}$ (see figure \ref{['fig:notationexample']}). The number next to edge shows the comm step using the edge. Figure (b) shows the expanded topology. Observe that every edge in $K_{2,2}$ becomes a vertex in $L(K_{2,2})$, and two vertices are connected if the corresponding edges in $K_{2,2}$ have one's head node being the other's tail node. Figure (c) shows the broadcast paths of node $ca$, transformed from the broadcast paths of $a$ in figure (a). At the 1st comm step, by step 1 of def \ref{['def:linegraph']}, $ca$ broadcasts its shard to all its neighbors: $((ca,S),(ca,ac),1)$, $((ca,S),(ca,ad),1)$. The rest of the broadcast paths are transformed from $A_{K_{2,2}}$ by step 2 of def \ref{['def:linegraph']}, e.g. $((a,C_1),(c,b),2)\mapsto\{((ca,C_1),(cb,bc),3),((ca,C_1),(cb,bd),3)\}$. Each of the nodes $bc$ and $bd$ receives $C_1,C_2$ from its two in-neighbors, just like $b$ does in $A_{K_{2,2}}$.
• Figure 3: Line graph expansion on Moore and BW optimal degree-4 base graphs: complete bipartite graph $K_{4,4}$, complete graph, directed circulant graph, and Hamming graph $H(2,3)$. $T_B^*\!=\!\frac{M}{B}\!\cdot\!\frac{N-1}{N}$ is the optimal BW runtime.
• Figure 4: 4-node unidirectional ring and its degree expansion to $d\!=\!2$. Figure (a) shows the base topology and broadcast path from $a$ to $b,c,d$. Figure (b) shows the expanded topology. Figure (c) shows the broadcast paths from $a_1$ and $a_2$ to other nodes marked in red and blue respectively. For any $u\neq a$, the path from $a_i$ to $u_j$ stays in $i$ until the very last step when it jumps to $u_j$, e.g., $a_1\!\!\mathrel{ \vcenter{} \mkern-4mu\hbox{)}}\!\! b_1\!\!\mathrel{ \vcenter{} \mkern-4mu\hbox{)}}\!\! c_2$ and $a_2\!\!\mathrel{ \vcenter{} \mkern-4mu\hbox{)}}\!\! b_2\!\!\mathrel{ \vcenter{} \mkern-4mu\hbox{)}}\!\! c_2\!\!\mathrel{ \vcenter{} \mkern-4mu\hbox{)}}\!\! d_1$. For $a_i$ to $a_j$, each in-neighbor of $a_j$ is tasked with sending an equal portion of $a_i$'s shard to $a_j$ in the end; for example, $d_1$ and $d_2$ each send half of $a_1$'s shard to $a_2$ and also each send half of $a_2$'s shard to $a_1$. The broadcast paths are disjoint from each other, resulting in BW optimality.
• Figure 5: Example of BFB allgather schedule at comm step $t$. Here, $w_1,w_2\in N^+_{t-1}(v_1)$ and $w_2,w_3\in N^+_{t-1}(v_2)$. $u_1,u_2$ are at distance $t$ from both $v_1,v_2$, so they both need to receive the data shards of $v_1,v_2$ in comm step $t$. Note that $u_1$ cannot get $v_2$'s shard from $w_3$ because $w_3$ is not an in-neighbor of $u_1$. The figure also shows the solutions to LPs (\ref{['lpmodel']}). The red and blue are independent LPs optimizing $U_{u_1,t}$ and $U_{u_2,t}$ respectively.

Theorems & Definitions (106)

• Definition 1: Schedule of Line Graph
• Definition 2: Degree Expanded Schedule
• Definition 3: Cartesian Product
• Definition 4: Allgather
• Definition 5: Reverse Schedule
• Theorem 1
• Corollary 1.1
• Corollary 1.2
• Definition 6: Reverse-Symmetry
• Definition 7: Schedule Isomorphism