Rina: Enhancing Ring-AllReduce with In-network Aggregation in Distributed Model Training

TL;DR

This paper tackles synchronization bottlenecks in Distributed Deep Learning by integrating In-Network Aggregation (INA) into Ring-AllReduce (RAR) through an agent–worker architecture, yielding substantial throughput gains and improved incremental deployment. It introduces the Bandwidth-Occupation Model (BOM) to analyze PS-based INA and demonstrates its poor incremental deployment, motivating a shift to INA-enabled RAR. The proposed Rina design compresses intra-rack dependencies, leverages INA-enabled switches for ScatterReduce and AllGather, and maintains compatibility with regular RAR workloads. Through extensive NS3 simulations and a real 8-node testbed, Rina outperforms PS, RAR, H-AR, and PS-based INA, achieving up to 50% higher throughput with partial INA deployment and comparable results to state-of-the-art INA methods when fully deployed, signaling practical benefits for data-center DDL workloads.

Abstract

Parameter Server (PS) and Ring-AllReduce (RAR) are two widely utilized synchronization architectures in multi-worker Deep Learning (DL), also referred to as Distributed Deep Learning (DDL). However, PS encounters challenges with the ``incast'' issue, while RAR struggles with problems caused by the long dependency chain. The emerging In-network Aggregation (INA) has been proposed to integrate with PS to mitigate its incast issue. However, such PS-based INA has poor incremental deployment abilities as it requires replacing all the switches to show significant performance improvement, which is not cost-effective. In this study, we present the incorporation of INA capabilities into RAR, called RAR with In-Network Aggregation (Rina), to tackle both the problems above. Rina features its agent-worker mechanism. When an INA-capable ToR switch is deployed, all workers in this rack run as one abstracted worker with the help of the agent, resulting in both excellent incremental deployment capabilities and better throughput. We conducted extensive testbed and simulation evaluations to substantiate the throughput advantages of Rina over existing DDL training synchronization structures. Compared with the state-of-the-art PS-based INA methods ATP, Rina can achieve more than 50\% throughput with the same hardware cost.

Figures (12)

• Figure 1: Popular synchronization architectures in DDL.
• Figure 2: Details for Ring-AllReduce.
• Figure 3: The node error will block the whole RAR synchronization process.
• Figure 4: A sample of traffic on the switch under PS architecture.
• Figure 5: Lack of incremental deployment capability for PS-based INA.