Enabling Reconfiguration-Communication Overlap for Collective Communication in Optical Networks

TL;DR

The paper addresses inefficiencies in large-scale distributed ML due to static, one-shot optical reconfiguration for CC. It proposes SWOT, a framework that overlaps OCS reconfiguration with data transmission through an offline MILP-based scheduler, a lightweight CC shim, and an optical controller, enabling dynamic alignment of optical resources with CC traffic. Key results show substantial reductions in CCT (up to ~74%) across multiple CC algorithms and improved scalability beyond traditional one-shot approaches. This co-adaptive, demand-aware approach can significantly improve the practicality of reconfigurable optical interconnects for modern DML workloads.

Abstract

Collective communication (CC) is widely adopted for large-scale distributed machine learning (DML) training workloads. DML's predictable traffic pattern provides a great oppotunity for applying optical network technology. Existing optical interconnects-based CC schemes adopt ``one-shot network reconfiguration'', which provisions static high-capacity topologies for an entire collective operation -- sometimes for a full training iteration. However, this approach faces significant scalability limitations when supporting more complex and efficient CC algorithms required for modern workloads: the ``one-shot'' strategies either demand excessive resource overprovisioning or suffer performance degradation due to rigid resource allocation. To address these challenges, we propose SWOT, a demand-aware optical network framework. SWOT employs ``intra-collective reconfiguration'' and can dynamically align network resources with CC traffic patterns. SWOT incorporates a novel scheduling technique that overlaps optical switch reconfigurations with ongoing transmissions, and improves communication efficiency. SWOT introduce a lightweight collective communication shim that enables coordinated optical network configuration and transmission scheduling while supporting seamless integration with existing CC libraries. Our simulation results demonstrate SWOT's significant performance improvements.

Figures (8)

• Figure 1: Example of OCS interconnect. Note that the nodes on the left side and right side are the same set of nodes; the left side denotes the TX path, and the right side denotes the RX path.
• Figure 2: Example of optical network topology, where OCS$_1$ is reconfigured matching Fig. \ref{['subfig: interconnect']}.
• Figure 3: Rabenseifner's algorithm for AllReduce with 8 nodes, where the collective size is 40 MB. It operates in a 6-step execution with 3 distinct bijective pairings.
• Figure 4: OCS configuration corresponding to traffic pattern in 8-node Rabenseifner's AllReduce (see Fig \ref{['fig: allreduce-hd']})
• Figure 5: Motivation example of reconfiguration-communication overlapping design. Communication-reconfiguration timeline for 8-node AllReduce in optical network: (a) naive intra-collective reconfiguration incurs cumulative $800$ µs switching overhead, (b) SWOT's overlap-optimized approach reduces CCT by 20% through partial circuit updates during transmissions (Illustrated as (c)). Configuration details ($400$Gbps links, $200$ µs reconfiguration delay), traffic patterns and required OCS configurations shown in Fig. \ref{['fig: allreduce-hd']} and Fig. \ref{['fig: mapping']}.