Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reordering

TL;DR

FlashOverlap addresses the inter-GPU communication bottleneck in multi-GPU generative-model workloads by introducing a signaling-based overlap that preserves GEMM performance while overlapping communication. The method leverages wave- and group-aware signaling, plus pre- and post-communication reordering, to enable NCCL-API-based communication without modifying core GEMM logic. A predictive real-time tuning (wave-group partitioning) and kernel-fusion strategy yields up to 1.65x overlap speedups and 1.05–1.13x end-to-end gains across inference and training in diverse tasks, with robust performance across AllReduce, ReduceScatter, and All-to-All primitives. The approach is hardware- and library-agnostic to a large extent (NVIDIA NCCL, CUDA/CUTLASS; also demonstrated on Huawei Ascend NPUs), offering practical impact for accelerating large-scale multi-GPU deployments with minimal development overhead.

Abstract

Generative models have achieved remarkable success across various applications, driving the demand for multi-GPU computing. Inter-GPU communication becomes a bottleneck in multi-GPU computing systems, particularly on consumer-grade GPUs. By exploiting concurrent hardware execution, overlapping computation and communication latency becomes an effective technique for mitigating the communication overhead. We identify that an efficient and adaptable overlapping design should satisfy (1) tile-wise overlapping to maximize the overlapping opportunity, (2) interference-free computation to maintain the original computational performance, and (3) communication agnosticism to reduce the development burden against varying communication primitives. Nevertheless, current designs fail to simultaneously optimize for all of those features. To address the issue, we propose FlashOverlap, which utilizes a novel signaling mechanism: when part of the output finishes, the computation kernel sends a signal to trigger the communication of that part, while continuing the computation of the remaining part (interference-free computation). Consequently, the communication of the finished part and the computation of the remaining part can be overlapped. On top of the signaling mechanism, FlashOverlap comprises two key components: (1) the determination of the signaling timing to boost the overlap efficiency (tile-wise overlapping), and (2) a pre-communication reordering to create the contiguous address for finished data, enabling communication by simply calling NCCL APIs (communication agnosticism), and a post-communication reordering to correct the data order. Experiments show that FlashOverlap achieves up to 1.65x speedup through overlap, outperforming existing works in most cases. Code is available at https://github.com/infinigence/FlashOverlap.

Figures (16)

• Figure 1: Overlapping methods. (1) Decomposition-based methods are easy to implement while yielding suboptimal overlapping efficiency, (2) fusion-based methods are efficient at the cost of high adaptation efforts, while (3) the proposed signaling-based method optimizes for both efficiency and easy adaptation.
• Figure 2: Tile partition and execution in GEMM.
• Figure 3: Wave pattern in GEMM execution. Each point in (a) and (b) represents the corresponding completion time of each tile, and the time is captured by the global timer globaltimer.
• Figure 4: Typical time portion of "GEMM + X" in inference and training. All profilings are on A800 GPUs.
• Figure 5: System overview. The GEMM computation is executed within one GPU kernel, and when each group ($G_1$, $G_2$, or $G_3$) of tiles finishes, it first reorders the tiles in the group to contiguous addresses, and then signals to trigger the corresponding inter-GPU communication of the group. To correct the order, the tiles are reordered back when communication finishes.