MSCCL++: Rethinking GPU Communication Abstractions for AI Inference

TL;DR

MSCCL++ addresses the persistent challenge of GPU communication bottlenecks in AI by introducing a three-layer design: a low-level Primitive API with PortChannel, MemoryChannel, and SwitchChannel abstractions; a Python-based DSL for workload-specific algorithm definitions; and a Collectives Library that mimics NCCL/RCCL APIs. This separation enables expert users to exploit hardware features for peak performance while providing higher-level interfaces for rapid development and broad adoption. The framework delivers substantial speedups in collective operations and end-to-end AI inference across NVIDIA and AMD platforms, including cross-hardware portability and rapid hardware feature adoption (e.g., multimem). By demonstrating close-to-hardware control, workload-aware algorithm design, and easy integration with existing AI pipelines, MSCCL++ promises to reduce development effort and accelerate deployment of high-performance AI workloads in production. The results suggest MSCCL++ can generalize beyond GPU-only setups to broader AI accelerators and IO pathways future-proofing interconnect optimization.

Abstract

AI applications increasingly run on fast-evolving, heterogeneous hardware to maximize performance, but general-purpose libraries lag in supporting these features. Performance-minded programmers often build custom communication stacks that are fast but error-prone and non-portable. This paper introduces MSCCL++, a design methodology for developing high-performance, portable communication kernels. It provides (1) a low-level, performance-preserving primitive interface that exposes minimal hardware abstractions while hiding the complexities of synchronization and consistency, (2) a higher-level DSL for application developers to implement workload-specific communication algorithms, and (3) a library of efficient algorithms implementing the standard collective API, enabling adoption by users with minimal expertise. Compared to state-of-the-art baselines, MSCCL++ achieves geomean speedups of $1.7\times$ (up to $5.4\times$) for collective communication and $1.2\times$ (up to $1.38\times$) for AI inference workloads. MSCCL++ is in production of multiple AI services provided by Microsoft Azure, and has also been adopted by RCCL, the GPU collective communication library maintained by AMD. MSCCL++ is open source and available at https://github.com/microsoft/mscclpp . Our two years of experience with MSCCL++ suggests that its abstractions are robust, enabling support for new hardware features, such as multimem, within weeks of development.

Figures (13)

• Figure 1: Overview of the MSCCL++ stack.
• Figure 2: Channel abstractions in MSCCL++ Primitive API.
• Figure 3: MSCCL++ data transfer abstractions. put asynchronously transfers data from one GPU to another. signal and wait synchronize data transfer between GPUs. flush ensures the completion of preceded data transfer.
• Figure 4: PortChannel workflow for IB from ⓪ when GPU 0 calls put primitive to ⑧ when GPU 1 receives the data.
• Figure 5: Channel abstractions in MSCCL++ DSL (Python).